# MCP CDP Scraper — Web Scraping via Chrome DevTools Protocol for AI Agents

> An MCP (Model Context Protocol) server that gives AI agents the ability to scrape web pages, fill forms, take screenshots, and extract structured data — all through Chrome DevTools Protocol with zero external dependencies.

## Why This Exists

Most web scraping MCP servers wrap Playwright or Puppeteer — heavy dependencies that break in Docker, require npm, and add 200MB+ to your image. This server uses **raw CDP over WebSockets** with Python stdlib only. It connects to any Chrome/Chromium instance running with `--remote-debugging-port` and gives your AI agent 12 scraping tools.

**No Playwright. No Selenium. No Puppeteer. No npm. Just Python stdlib + a running Chrome.**

## Features

- **12 MCP Tools**: `list_tabs`, `scrape_page`, `extract_text`, `extract_links`, `extract_images`, `extract_table`, `fill_form`, `click_element`, `screenshot`, `get_html`, `wait_for`, `scroll_to`
- **Zero dependencies** — pure Python stdlib (socket, json, urllib, base64, struct)
- **React/Vue compatible form filling** — uses native setters + synthetic events
- **STDIO JSON-RPC mode** — drop-in for any MCP-compatible client (Claude Desktop, Hermes, etc.)
- **Structured extraction** — title, meta, headings, paragraphs, lists, links, images in one call
- **Screenshot capture** — base64 PNG or save to file
- **Form automation** — fill fields, click submit, wait for dynamic content

## Quick Start

### 1. Launch Chrome with remote debugging

```bash
# Linux
google-chrome --remote-debugging-port=9222 --headless=new &

# macOS
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222 &

# Docker
docker run -d -p 9222:9222 chromedp/chrome --remote-debugging-port=9222 --headless
```

### 2. Run the MCP server

```bash
# STDIO mode (for MCP clients)
python -m src.server --stdio

# Or print the manifest
python -m src.server --manifest
```

### 3. Use as a library

```python
from src.server import MCPCDPScraperServer

server = MCPCDPScraperServer()

# Scrape a page
result = server.handle_tool_call("scrape_page", {"url": "https://news.ycombinator.com"})
import json
data = json.loads(result)
print(data["structured"]["title"])
print(f"Found {data['link_count']} links")

# Fill a form
result = server.handle_tool_call("fill_form", {
    "fields": {
        "#email": "user@example.com",
        "#password": "secret123"
    },
    "submit": "#login-button"
})
```

## MCP Tool Reference

| Tool | Description | Required Params |
|------|-------------|-----------------|
| `list_tabs` | List open browser tabs | — |
| `scrape_page` | Navigate + extract everything | `url` |
| `extract_text` | Get text from element | `selector` |
| `extract_links` | Get all links | — |
| `extract_images` | Get all images | — |
| `extract_table` | Get table as 2D array | `selector` |
| `fill_form` | Fill form fields (React-safe) | `fields` |
| `click_element` | Click by selector | `selector` |
| `screenshot` | Capture screenshot | — |
| `get_html` | Get element HTML | `selector` |
| `wait_for` | Wait for selector | `selector` |
| `scroll_to` | Scroll to element/coords | — |

## Integration with MCP Clients

### Claude Desktop (`claude_desktop_config.json`)

```json
{
  "mcpServers": {
    "cdp-scraper": {
      "command": "python",
      "args": ["-m", "src.server", "--stdio"],
      "cwd": "/path/to/mcp-cdp-scraper"
    }
  }
}
```

### Hermes Agent (config.yaml)

```yaml
mcp:
  servers:
    cdp-scraper:
      command: python
      args: ["-m", "src.server", "--stdio"]
      cwd: /path/to/mcp-cdp-scraper
```

## Demo

```bash
# Requires Chrome running on port 9222
python demo.py
```

## Tests

```bash
python -m pytest tests/ -v
```

## Docker

```bash
docker build -t mcp-cdp-scraper .
docker run -p 9222:9222 mcp-cdp-scraper
```

## Architecture

```
MCP Client (Claude/Hermes/any)
    │  JSON-RPC over STDIO
    ▼
MCPCDPScraperServer (src/server.py)
    │  Tool dispatch + schema
    ▼
CDPScraper (src/cdp_scrape.py)
    │  Raw WebSocket CDP commands
    ▼
Chrome/Chromium (--remote-debugging-port=9222)
    │
    ▼
Web Page → Structured Data
```

## Use Cases

- **AI agent web scraping** — Let your agent browse and extract data from any page
- **Form automation** — Login flows, signups, search submissions
- **Content monitoring** — Scrape and diff pages on a schedule
- **Data extraction pipelines** — Tables, product listings, article content
- **Screenshot generation** — Visual capture for QA or documentation
- **Anti-bot bypass** — Real Chrome instance avoids headless detection

## License

MIT

## Author

aaameobius-crypto — [github.com/aaameobius-crypto](https://github.com/aaameobius-crypto)

Freelance portfolio: [https://ameobius-space.github.io/kwork-portfolio/](https://ameobius-space.github.io/kwork-portfolio/)