Selenium MCP Server
Allows web crawling and data extraction using Selenium WebDriver, supporting CSS/XPath selectors, pagination, infinite scroll, screenshots, and more.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
🕷️ Selenium MCP Server
MCP server (stdio) sử dụng Selenium để crawl và trích xuất dữ liệu từ web pages bằng CSS/XPath selectors.
✨ Tính Năng
MCP Tools (17 tools)
Tool | Mô Tả |
| Trích xuất dữ liệu cơ bản với CSS/XPath selectors |
| Phân tích cấu trúc trang trước khi extract |
| Tự động detect loại trang và extract (products, article, all) |
| Crawl nhiều trang với pagination |
| Xử lý lazy loading/infinite scroll |
| Khám phá sitemap, categories, navigation |
| Crawl nhiều category URLs |
| Crawl listing + chi tiết từng sản phẩm |
| Extract chi tiết từ danh sách URLs |
| Xuất CSV/JSON |
| Chụp full-page screenshot |
| Chụp screenshot của element cụ thể |
| Get/Set/Clear browser cookies |
| Đăng nhập và extract từ trang protected |
| Thực thi JavaScript tùy chỉnh |
| Lấy performance metrics của trang |
REST API Endpoints
Server cũng cung cấp REST API đầy đủ qua FastAPI:
POST /extract- Basic extractionPOST /smart-extract- Smart extractionPOST /analyze- Analyze pagePOST /crawl/pages- Crawl with paginationPOST /crawl/infinite-scroll- Handle infinite scrollPOST /crawl/categories- Crawl categoriesPOST /crawl/deep- Deep crawlPOST /discover- Discover sitePOST /products/details- Product detailsPOST /screenshot- Capture screenshotPOST /export- Export dataGET /jobs- List jobsGET /outputs- List output files
Related MCP server: rubber-duck-scraper
🚀 Cài Đặt
# Clone repo
git clone https://github.com/Tai-DT/mcp-crawler.git
cd mcp-crawler
# Tạo virtual environment
python3 -m venv venv
source venv/bin/activate # Linux/Mac
# hoặc: venv\Scripts\activate # Windows
# Cài đặt dependencies
pip install -r requirements.txt📦 Chạy Server
MCP Server (cho AI clients)
# Stdio mode (mặc định, cho MCP clients)
python3 selenium_mcp_server.py
# HTTP mode (cho debugging)
python3 selenium_mcp_server.py --transport streamable-http --host 127.0.0.1 --port 8000REST API Server
python3 server_api.py
# API sẽ chạy tại http://localhost:8000Docker
docker-compose up -d
# API: http://localhost:8000⚙️ Cấu Hình MCP Client
VS Code (mcp.json)
{
"servers": {
"local/selenium-crawler": {
"type": "stdio",
"command": "/path/to/mcp-crawler/venv/bin/python",
"args": ["/path/to/mcp-crawler/selenium_mcp_server.py"],
"env": {}
}
}
}Claude Desktop
{
"mcpServers": {
"selenium-crawler": {
"command": "python3",
"args": ["/path/to/mcp-crawler/selenium_mcp_server.py"]
}
}
}📖 Ví Dụ Sử Dụng
Basic Extract
{
"url": "https://example.com",
"fields": {
"title": "css:h1",
"description": "css:p",
"links": {"selector": "css:a", "attribute": "href", "multiple": true}
}
}Smart Extract Products
{
"url": "https://shop.example.com/category",
"extract_type": "products",
"limit": 50,
"normalize_prices": true
}Deep Crawl
{
"url": "https://shop.example.com/products",
"follow_links": "products",
"max_links": 20,
"normalize_prices": true
}Screenshot
{
"url": "https://example.com",
"full_page": true,
"width": 1920,
"height": 1080
}Login and Extract
{
"login_url": "https://example.com/login",
"target_url": "https://example.com/dashboard",
"username_selector": "css:#email",
"password_selector": "css:#password",
"submit_selector": "css:button[type='submit']",
"username": "user@example.com",
"password": "password123",
"fields": {
"welcome_message": "css:.welcome",
"account_name": "css:.account-name"
}
}🔄 Retry & Error Handling
Server có built-in retry với exponential backoff:
Max Retries: 3 lần
Initial Delay: 1 giây
Backoff Factor: 2x
Rate Limiting: 30 requests/phút
📁 Cấu Trúc Dự Án
mcp-crawler/
├── selenium_mcp_server.py # MCP Server chính (17 tools)
├── server_api.py # REST API (FastAPI)
├── crawler_engine.py # Crawler engine core
├── extraction_utils.py # Tiện ích xử lý dữ liệu
├── config.py # Configuration management
├── retry_handler.py # Retry & rate limiting
├── job_manager.py # Job tracking & persistence
├── requirements.txt # Python dependencies
├── Dockerfile # Docker image
├── docker-compose.yml # Docker Compose
├── outputs/ # Extracted data
│ └── screenshots/ # Captured screenshots
├── jobs/ # Job history
└── scripts/ # Test scripts🧪 Testing
# Test crawler engine
python3 test_mcp.py
# Smoke test MCP server
python3 scripts/smoke_test_stdio.py📝 License
MIT
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
Web scraping MCP server — scrape, extract structured data, screenshot any site with anti-bot bypass.
MCP server for web extraction and rendering via AceDataCloud WebExtrator
All HasData scraping tools in one MCP server: Google, TikTok, Instagram, maps, e-commerce and more.
Web scraping, browser automation, screenshots and AI data extraction for AI agents via MCP.
Related MCP Servers
- AlicenseAqualityCmaintenanceMCP server for web crawling, searching, and AI-powered content extraction, supporting single-page, batch, and full-site crawling along with text, news, image, book, and video search.81MIT
- FlicenseNot gradedqualityDmaintenancePython MCP server that scrapes web pages with JS rendering, structured metadata, tables, PDFs, screenshots, and multi-page crawling.-
- AlicenseNot gradedqualityDmaintenanceA general-purpose MCP server for crawling and extracting structured data from any website. Supports tools for crawling, single-page extraction, search-and-crawl, and schema extraction.2MIT
- AlicenseCqualityCmaintenanceMCP server for browser automation that drives Chrome via an extension, preserving login state and offering 45 tools for navigation, interaction, scraping, and screenshots.535MIT