Skip to main content
Glama
Tai-DT

Selenium MCP Server

by Tai-DT
OverviewSchemaRelated ServersScoreDiscussions
Python
Remote

🕷️ Selenium MCP Server

MCP server (stdio) sử dụng Selenium để crawl và trích xuất dữ liệu từ web pages bằng CSS/XPath selectors.

✨ Tính Năng

MCP Tools (17 tools)

Tool

Mô Tả

extract

Trích xuất dữ liệu cơ bản với CSS/XPath selectors

analyze_page

Phân tích cấu trúc trang trước khi extract

smart_extract

Tự động detect loại trang và extract (products, article, all)

crawl_all_pages

Crawl nhiều trang với pagination

crawl_infinite_scroll

Xử lý lazy loading/infinite scroll

discover_site

Khám phá sitemap, categories, navigation

crawl_category_list

Crawl nhiều category URLs

deep_crawl

Crawl listing + chi tiết từng sản phẩm

extract_product_details

Extract chi tiết từ danh sách URLs

export_results

Xuất CSV/JSON

take_screenshot

Chụp full-page screenshot

take_element_screenshot

Chụp screenshot của element cụ thể

manage_cookies

Get/Set/Clear browser cookies

login_and_extract

Đăng nhập và extract từ trang protected

execute_javascript

Thực thi JavaScript tùy chỉnh

get_page_metrics

Lấy performance metrics của trang

REST API Endpoints

Server cũng cung cấp REST API đầy đủ qua FastAPI:

  • POST /extract - Basic extraction

  • POST /smart-extract - Smart extraction

  • POST /analyze - Analyze page

  • POST /crawl/pages - Crawl with pagination

  • POST /crawl/infinite-scroll - Handle infinite scroll

  • POST /crawl/categories - Crawl categories

  • POST /crawl/deep - Deep crawl

  • POST /discover - Discover site

  • POST /products/details - Product details

  • POST /screenshot - Capture screenshot

  • POST /export - Export data

  • GET /jobs - List jobs

  • GET /outputs - List output files

Related MCP server: Site Crawler MCP

🚀 Cài Đặt

# Clone repo
git clone https://github.com/Tai-DT/mcp-crawler.git
cd mcp-crawler

# Tạo virtual environment
python3 -m venv venv
source venv/bin/activate  # Linux/Mac
# hoặc: venv\Scripts\activate  # Windows

# Cài đặt dependencies
pip install -r requirements.txt

📦 Chạy Server

MCP Server (cho AI clients)

# Stdio mode (mặc định, cho MCP clients)
python3 selenium_mcp_server.py

# HTTP mode (cho debugging)
python3 selenium_mcp_server.py --transport streamable-http --host 127.0.0.1 --port 8000

REST API Server

python3 server_api.py
# API sẽ chạy tại http://localhost:8000

Docker

docker-compose up -d
# API: http://localhost:8000

⚙️ Cấu Hình MCP Client

VS Code (mcp.json)

{
  "servers": {
    "local/selenium-crawler": {
      "type": "stdio",
      "command": "/path/to/mcp-crawler/venv/bin/python",
      "args": ["/path/to/mcp-crawler/selenium_mcp_server.py"],
      "env": {}
    }
  }
}

Claude Desktop

{
  "mcpServers": {
    "selenium-crawler": {
      "command": "python3",
      "args": ["/path/to/mcp-crawler/selenium_mcp_server.py"]
    }
  }
}

📖 Ví Dụ Sử Dụng

Basic Extract

{
  "url": "https://example.com",
  "fields": {
    "title": "css:h1",
    "description": "css:p",
    "links": {"selector": "css:a", "attribute": "href", "multiple": true}
  }
}

Smart Extract Products

{
  "url": "https://shop.example.com/category",
  "extract_type": "products",
  "limit": 50,
  "normalize_prices": true
}

Deep Crawl

{
  "url": "https://shop.example.com/products",
  "follow_links": "products",
  "max_links": 20,
  "normalize_prices": true
}

Screenshot

{
  "url": "https://example.com",
  "full_page": true,
  "width": 1920,
  "height": 1080
}

Login and Extract

{
  "login_url": "https://example.com/login",
  "target_url": "https://example.com/dashboard",
  "username_selector": "css:#email",
  "password_selector": "css:#password",
  "submit_selector": "css:button[type='submit']",
  "username": "user@example.com",
  "password": "password123",
  "fields": {
    "welcome_message": "css:.welcome",
    "account_name": "css:.account-name"
  }
}

🔄 Retry & Error Handling

Server có built-in retry với exponential backoff:

  • Max Retries: 3 lần

  • Initial Delay: 1 giây

  • Backoff Factor: 2x

  • Rate Limiting: 30 requests/phút

📁 Cấu Trúc Dự Án

mcp-crawler/
├── selenium_mcp_server.py  # MCP Server chính (17 tools)
├── server_api.py           # REST API (FastAPI)
├── crawler_engine.py       # Crawler engine core
├── extraction_utils.py     # Tiện ích xử lý dữ liệu
├── config.py               # Configuration management
├── retry_handler.py        # Retry & rate limiting
├── job_manager.py          # Job tracking & persistence
├── requirements.txt        # Python dependencies
├── Dockerfile              # Docker image
├── docker-compose.yml      # Docker Compose
├── outputs/                # Extracted data
│   └── screenshots/        # Captured screenshots
├── jobs/                   # Job history
└── scripts/                # Test scripts

🧪 Testing

# Test crawler engine
python3 test_mcp.py

# Smoke test MCP server
python3 scripts/smoke_test_stdio.py

📝 License

MIT

This server cannot be installed

F
license - not found
-
quality - not tested
D
maintenance

How are these scores calculated?

Maintenance

–Maintainers
–Response time
–Release cycle
–Releases (12mo)
Commit activity

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/Tai-DT/mcp-crawler'

If you have feedback or need assistance with the MCP directory API, please join our Discord server