Web Scraper MCP

A self-hosted Model Context Protocol server that gives an LLM client (Claude Code, Cursor, ChatGPT) the same tool surface as paid scraping services — scrape, crawl, map, search, extract, interact, deep_research — running entirely on your own hardware.

No paid proxy/CAPTCHA services: anti-bot is self-hosted (headless Chromium + playwright-stealth, robots.txt, polite rate limiting). Hardened sites may still block; see Limitations.

Tools

extract and deep_research require ANTHROPIC_API_KEY.

Related MCP server: Universal Web Data Extraction Platform

Quick start

uv sync                      # install deps (uses the pinned uv.lock)
uv run playwright install chromium
export SCRAPER_AUTH_TOKEN=$(openssl rand -hex 32)
uv run web-scraper-mcp       # HTTP server on http://127.0.0.1:8000/mcp

Stdio (local, for a desktop client): SCRAPER_TRANSPORT=stdio uv run web-scraper-mcp.

Docker

The fastest way to get started is by pulling the pre-built image from DockerHub.

# Pull the latest image
docker pull PROG_UP_USERNAME/web-scraper-mcp:latest

# Run the container (with Anthropic / Claude)
docker run -p 8000:8000 \
  -e SCRAPER_AUTH_TOKEN=your_secure_token_here \
  -e ANTHROPIC_API_KEY=sk-ant-api03-... \
  PROG_UP_USERNAME/web-scraper-mcp:latest

# Or run the container over stdio (useful for local MCP clients)
docker run -i --rm \
  -e SCRAPER_TRANSPORT=stdio \
  -e SCRAPER_AUTH_TOKEN=your_secure_token_here \
  PROG_UP_USERNAME/web-scraper-mcp:latest

(Make sure to replace PROG_UP_USERNAME with your actual DockerHub username).

Using Local Models (Ollama) & Context Windows

If you prefer to run models locally instead of using Anthropic's API, the server fully supports Ollama as an alternative backend for the extract and deep_research tools.

docker run -p 8000:8000 \
  -e SCRAPER_AUTH_TOKEN=your_secure_token_here \
  -e SCRAPER_OLLAMA_HOST=http://host.docker.internal:11434 \
  -e SCRAPER_EXTRACT_MODEL=qwen3.5:2b \
  -e SCRAPER_RESEARCH_MODEL=qwen3.5:2b \
  PROG_UP_USERNAME/web-scraper-mcp:latest

Register in a client (mcp.json)

If running via HTTP:

{
  "mcpServers": {
    "web-scraper": {
      "url": "http://127.0.0.1:8000/mcp",
      "headers": { "Authorization": "Bearer your_secure_token_here" }
    }
  }
}

If running via stdio (Docker):

{
  "mcpServers": {
    "web-scraper": {
      "command": "docker",
      "args": [
        "run", "-i", "--rm",
        "-e", "SCRAPER_TRANSPORT=stdio",
        "-e", "SCRAPER_OLLAMA_HOST=http://host.docker.internal:11434",
        "-e", "SCRAPER_EXTRACT_MODEL=qwen3.5:2b",
        "-e", "SCRAPER_RESEARCH_MODEL=qwen3.5:2b",
        "PROG_UP_USERNAME/web-scraper-mcp:latest"
      ]
    }
  }
}

Configuration

All settings are env vars (prefix SCRAPER_), or a .env file — see .env.example.

Security

• SSRF guard — every fetched URL (and each redirect hop) is DNS-resolved and rejected if it points at a private / loopback / link-local / cloud-metadata address. The browser also aborts subresource requests to private IPs.

• Auth — bearer token on the HTTP transport; bind localhost by default.

• Resource caps — response-size, timeout, page-concurrency, crawl page/depth limits to protect the host.

• robots.txt + rate limiting on by default.

• Container — runs as a non-root user; secrets via env only.

Supply chain — verifying the image

CI signs the image keylessly with cosign (Sigstore) using GitLab's OIDC identity, and attaches an SPDX SBOM attestation. Verify before running:

cosign verify \
  --certificate-oidc-issuer https://gitlab.cri.epita.fr \
  --certificate-identity-regexp 'https://gitlab.cri.epita.fr/enzo.juhel/web-scraper//.*' \
  registry.gitlab.cri.epita.fr/enzo.juhel/web-scraper@sha256:...

Benchmark

benchmarks/run.py scores our scrape/extract against public datasets and the Crawl4AI baseline, emitting a scorecard (per page-type F1/accuracy + a Limitations section). Run locally or via the manual CI benchmark job:

uv run python benchmarks/run.py --output scorecard.md

Limitations

• No paid proxies/CAPTCHA: heavily-defended sites (LinkedIn, Amazon, Cloudflare challenges) will sometimes block us. The benchmark scorecard quantifies where.

• Main-content extraction is strong on articles, weaker on forums / product / listing pages (a known property of all extractors).

• In-memory crawl/session state — single-process, single-user by design.

Development

uv run pre-commit install        # local lint/secret hooks
uv run ruff check . && uv run ruff format --check .
uv run mypy src
uv run pytest -q