MCP CDP Scraper — Web Scraping via Chrome DevTools Protocol for AI Agents

An MCP (Model Context Protocol) server that gives AI agents the ability to scrape web pages, fill forms, take screenshots, and extract structured data — all through Chrome DevTools Protocol with zero external dependencies.

Why This Exists

Most web scraping MCP servers wrap Playwright or Puppeteer — heavy dependencies that break in Docker, require npm, and add 200MB+ to your image. This server uses raw CDP over WebSockets with Python stdlib only. It connects to any Chrome/Chromium instance running with --remote-debugging-port and gives your AI agent 12 scraping tools.

No Playwright. No Selenium. No Puppeteer. No npm. Just Python stdlib + a running Chrome.

Features

• 12 MCP Tools: list_tabs, scrape_page, extract_text, extract_links, extract_images, extract_table, fill_form, click_element, screenshot, get_html, wait_for, scroll_to

• Zero dependencies — pure Python stdlib (socket, json, urllib, base64, struct)

• React/Vue compatible form filling — uses native setters + synthetic events

• STDIO JSON-RPC mode — drop-in for any MCP-compatible client (Claude Desktop, Hermes, etc.)

• Structured extraction — title, meta, headings, paragraphs, lists, links, images in one call

• Screenshot capture — base64 PNG or save to file

• Form automation — fill fields, click submit, wait for dynamic content

Quick Start

1. Launch Chrome with remote debugging

# Linux
google-chrome --remote-debugging-port=9222 --headless=new &

# macOS
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222 &

# Docker
docker run -d -p 9222:9222 chromedp/chrome --remote-debugging-port=9222 --headless

2. Run the MCP server

# STDIO mode (for MCP clients)
python -m src.server --stdio

# Or print the manifest
python -m src.server --manifest

3. Use as a library

from src.server import MCPCDPScraperServer

server = MCPCDPScraperServer()

# Scrape a page
result = server.handle_tool_call("scrape_page", {"url": "https://news.ycombinator.com"})
import json
data = json.loads(result)
print(data["structured"]["title"])
print(f"Found {data['link_count']} links")

# Fill a form
result = server.handle_tool_call("fill_form", {
    "fields": {
        "#email": "user@example.com",
        "#password": "secret123"
    },
    "submit": "#login-button"
})

MCP Tool Reference

Integration with MCP Clients

Claude Desktop (claude_desktop_config.json)

{
  "mcpServers": {
    "cdp-scraper": {
      "command": "python",
      "args": ["-m", "src.server", "--stdio"],
      "cwd": "/path/to/mcp-cdp-scraper"
    }
  }
}

Hermes Agent (config.yaml)

mcp:
  servers:
    cdp-scraper:
      command: python
      args: ["-m", "src.server", "--stdio"]
      cwd: /path/to/mcp-cdp-scraper

Demo

# Requires Chrome running on port 9222
python demo.py

Tests

python -m pytest tests/ -v

Docker

docker build -t mcp-cdp-scraper .
docker run -p 9222:9222 mcp-cdp-scraper

Architecture

MCP Client (Claude/Hermes/any)
    │  JSON-RPC over STDIO
    ▼
MCPCDPScraperServer (src/server.py)
    │  Tool dispatch + schema
    ▼
CDPScraper (src/cdp_scrape.py)
    │  Raw WebSocket CDP commands
    ▼
Chrome/Chromium (--remote-debugging-port=9222)
    │
    ▼
Web Page → Structured Data

Use Cases

• AI agent web scraping — Let your agent browse and extract data from any page

• Form automation — Login flows, signups, search submissions

• Content monitoring — Scrape and diff pages on a schedule

• Data extraction pipelines — Tables, product listings, article content

• Screenshot generation — Visual capture for QA or documentation

• Anti-bot bypass — Real Chrome instance avoids headless detection

License

MIT

Author

AMEOBIUS — github.com/AMEOBIUS