Skip to main content
Glama
Tai-DT

Selenium MCP Server

by Tai-DT
OverviewSchemaSourceRelated ServersDiscussionsAdmin
Python
Remote

🕷️ Selenium MCP Server

MCP server (stdio) sử dụng Selenium để crawl và trích xuất dữ liệu từ web pages bằng CSS/XPath selectors.

✨ Tính Năng

MCP Tools (17 tools)

Tool

Mô Tả

extract

Trích xuất dữ liệu cơ bản với CSS/XPath selectors

analyze_page

Phân tích cấu trúc trang trước khi extract

smart_extract

Tự động detect loại trang và extract (products, article, all)

crawl_all_pages

Crawl nhiều trang với pagination

crawl_infinite_scroll

Xử lý lazy loading/infinite scroll

discover_site

Khám phá sitemap, categories, navigation

crawl_category_list

Crawl nhiều category URLs

deep_crawl

Crawl listing + chi tiết từng sản phẩm

extract_product_details

Extract chi tiết từ danh sách URLs

export_results

Xuất CSV/JSON

take_screenshot

Chụp full-page screenshot

take_element_screenshot

Chụp screenshot của element cụ thể

manage_cookies

Get/Set/Clear browser cookies

login_and_extract

Đăng nhập và extract từ trang protected

execute_javascript

Thực thi JavaScript tùy chỉnh

get_page_metrics

Lấy performance metrics của trang

REST API Endpoints

Server cũng cung cấp REST API đầy đủ qua FastAPI:

  • POST /extract - Basic extraction

  • POST /smart-extract - Smart extraction

  • POST /analyze - Analyze page

  • POST /crawl/pages - Crawl with pagination

  • POST /crawl/infinite-scroll - Handle infinite scroll

  • POST /crawl/categories - Crawl categories

  • POST /crawl/deep - Deep crawl

  • POST /discover - Discover site

  • POST /products/details - Product details

  • POST /screenshot - Capture screenshot

  • POST /export - Export data

  • GET /jobs - List jobs

  • GET /outputs - List output files

Related MCP server: rubber-duck-scraper

🚀 Cài Đặt

# Clone repo
git clone https://github.com/Tai-DT/mcp-crawler.git
cd mcp-crawler

# Tạo virtual environment
python3 -m venv venv
source venv/bin/activate  # Linux/Mac
# hoặc: venv\Scripts\activate  # Windows

# Cài đặt dependencies
pip install -r requirements.txt

📦 Chạy Server

MCP Server (cho AI clients)

# Stdio mode (mặc định, cho MCP clients)
python3 selenium_mcp_server.py

# HTTP mode (cho debugging)
python3 selenium_mcp_server.py --transport streamable-http --host 127.0.0.1 --port 8000

REST API Server

python3 server_api.py
# API sẽ chạy tại http://localhost:8000

Docker

docker-compose up -d
# API: http://localhost:8000

⚙️ Cấu Hình MCP Client

VS Code (mcp.json)

{
  "servers": {
    "local/selenium-crawler": {
      "type": "stdio",
      "command": "/path/to/mcp-crawler/venv/bin/python",
      "args": ["/path/to/mcp-crawler/selenium_mcp_server.py"],
      "env": {}
    }
  }
}

Claude Desktop

{
  "mcpServers": {
    "selenium-crawler": {
      "command": "python3",
      "args": ["/path/to/mcp-crawler/selenium_mcp_server.py"]
    }
  }
}

📖 Ví Dụ Sử Dụng

Basic Extract

{
  "url": "https://example.com",
  "fields": {
    "title": "css:h1",
    "description": "css:p",
    "links": {"selector": "css:a", "attribute": "href", "multiple": true}
  }
}

Smart Extract Products

{
  "url": "https://shop.example.com/category",
  "extract_type": "products",
  "limit": 50,
  "normalize_prices": true
}

Deep Crawl

{
  "url": "https://shop.example.com/products",
  "follow_links": "products",
  "max_links": 20,
  "normalize_prices": true
}

Screenshot

{
  "url": "https://example.com",
  "full_page": true,
  "width": 1920,
  "height": 1080
}

Login and Extract

{
  "login_url": "https://example.com/login",
  "target_url": "https://example.com/dashboard",
  "username_selector": "css:#email",
  "password_selector": "css:#password",
  "submit_selector": "css:button[type='submit']",
  "username": "user@example.com",
  "password": "password123",
  "fields": {
    "welcome_message": "css:.welcome",
    "account_name": "css:.account-name"
  }
}

🔄 Retry & Error Handling

Server có built-in retry với exponential backoff:

  • Max Retries: 3 lần

  • Initial Delay: 1 giây

  • Backoff Factor: 2x

  • Rate Limiting: 30 requests/phút

📁 Cấu Trúc Dự Án

mcp-crawler/
├── selenium_mcp_server.py  # MCP Server chính (17 tools)
├── server_api.py           # REST API (FastAPI)
├── crawler_engine.py       # Crawler engine core
├── extraction_utils.py     # Tiện ích xử lý dữ liệu
├── config.py               # Configuration management
├── retry_handler.py        # Retry & rate limiting
├── job_manager.py          # Job tracking & persistence
├── requirements.txt        # Python dependencies
├── Dockerfile              # Docker image
├── docker-compose.yml      # Docker Compose
├── outputs/                # Extracted data
│   └── screenshots/        # Captured screenshots
├── jobs/                   # Job history
└── scripts/                # Test scripts

🧪 Testing

# Test crawler engine
python3 test_mcp.py

# Smoke test MCP server
python3 scripts/smoke_test_stdio.py

📝 License

MIT

This server cannot be deployed

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

Maintenance

ActivityInactive
ResponsivenessNo issues

Resources

Related MCP Connectors

Related MCP Servers

  • crawl-mcp

    gqy20
    A
    license
    A
    quality
    C
    maintenance
    MCP server for web crawling, searching, and AI-powered content extraction, supporting single-page, batch, and full-site crawling along with text, news, image, book, and video search.
    8
    2
    MIT

  • Scrapy MCP

    B1unt-alt
    A
    license
    Not graded
    quality
    C
    maintenance
    A general-purpose MCP server for crawling and extracting structured data from any website. Supports tools for crawling, single-page extraction, search-and-crawl, and schema extraction.
    2
    MIT

  • nekoro-browser-mcp

    zeshuochen
    A
    license
    C
    quality
    A
    maintenance
    MCP server for browser automation that drives Chrome via an extension, preserving login state and offering 45 tools for navigation, interaction, scraping, and screenshots.
    53
    5
    MIT