Crawl4AI Complete MCP

Floating latest-stable Docker deployment of Crawl4AI with the official MCP Python SDK stdio and Streamable HTTP servers. Builds resolve exact releases and source commits into generated metadata; manual build overrides are available only for rollback/debugging. The bridge does not expose arbitrary Python, shell, or HTTP proxying.

Status

The current verified build resolves Crawl4AI 0.9.2 (7e801521428ee12509994d39151006f64055ebe3) and MCP Python SDK 2.0.0 (6f69a3758ebf2ee55ce050f58b470ce11af71133). These are build evidence, not permanent pins. The inspected source remains the implementation authority for each build. The upstream deploy/docker tree was inspected before implementation; its HTTP server is a separate FastAPI process with Redis-backed jobs and monitoring. This project provides a stdio-first runtime and a safe, typed subset/bridge of the same local Python capability surface, with unsupported operations explicitly reported rather than silently discarded.

Related MCP server: servo-fetch

Quick start

docker compose build crawl4ai-mcp-stdio
# equivalent profile-aware form:
docker compose --profile stdio build crawl4ai-mcp-stdio
docker compose --profile stdio run --rm crawl4ai-mcp-stdio

For the separated native/Redis/HTTP topology, choose an unused host port instead of assuming 8000:

CRAWL4AI_MCP_HTTP_PORT=37600 docker compose --profile redis --profile native-api --profile stdio --profile http-mcp up -d

The container speaks MCP JSON-RPC over stdin/stdout. Logs go to stderr. Configure CRAWL4AI_API_TOKEN only if your MCP host requires an application token; it is not printed or returned.

Security defaults

• only http and https URLs;

• private, loopback, link-local, and cloud-metadata destinations blocked by default;

• optional domain allow-list;

• bounded URLs, scripts, output, timeout, concurrency, and crawl count;

• no arbitrary Python, shell, filesystem paths, Docker socket, or CAPTCHA/access-control bypass;

• declarative hooks only, disabled by default unless CRAWL4AI_HOOKS_ENABLED=true;

• proxy credentials must be supplied through environment-backed references, never MCP arguments;

• webhook delivery is not enabled in stdio mode (use the native HTTP job service for operator-managed webhooks).

See docs/security.md, docs/configuration.md, and docs/deployment.md.

MCP

The complete tool inventory and parity claims are in docs/mcp-tools.md and docs/parity-matrix.md. The bridge exposes individual tools for crawling, streaming, rendering, JavaScript, extraction, configuration and capability inventory. It is not an arbitrary HTTP proxy: operations are explicit and documented. It does not claim that Redis jobs, webhooks, native monitoring, or the upstream artefact store are present in a single stdio process; those are documented as separate authenticated HTTP deployment operations.

Native HTTP server

The pinned upstream release includes deploy/docker/server.py, a separate FastAPI service with /health, /schema, /crawl, /crawl/stream, /md, /html, /screenshot, /pdf, /execute_js, /hooks/info, job routes, monitoring routes, artefacts, and MCP SSE/WebSocket routes. It is documented in docs/upstream-api.md. This image deliberately does not start a hidden background HTTP process alongside stdio; run the upstream HTTP deployment separately when REST, SSE, Redis jobs, or Prometheus monitoring are required.

Development

uv sync --locked
uv run pytest -q
uv run python -m crawl4ai_mcp.server < /dev/null

Docker integration requires Docker and a working Chromium sandbox configuration. See docs/testing.md.

Licence

Apache-2.0. Crawl4AI is used under its upstream Apache-2.0 licence.