crawl4ai-mcp

A minimal MCP server for agent-friendly web extraction and search.

Two tools: fetch real pages with Playwright + Crawl4AI, or search across 7 engines with automatic fallback.

Quick entry

Related MCP server: web-mcp-server

At a glance

How it works

Fetch flow:

---
config:
  theme: base
  themeVariables:
    primaryColor: "#FAF9F5"
    primaryTextColor: "#1A1A1A"
    primaryBorderColor: "#D97757"
    lineColor: "#8B5E3C"
    secondaryColor: "#F5F1E8"
    tertiaryColor: "#FAF9F5"
    fontFamily: "ui-sans-serif, system-ui, -apple-system, sans-serif"
    fontSize: "14px"
---
flowchart LR
    A[URL list] --> B[Playwright + Crawl4AI]
    B --> C{Fast path enough?}
    C -- Yes --> D[Markdown / HTML]
    C -- No --> E[Stronger fallback]
    E --> D
    D --> F{use_llm?}
    F -- No --> G[Return result]
    F -- Yes --> H[OpenAI-compatible cleanup]
    H --> I{LLM success?}
    I -- Yes --> J[Return enhanced result]
    I -- No --> G

Search flow:

---
config:
  theme: base
  themeVariables:
    primaryColor: "#FAF9F5"
    primaryTextColor: "#1A1A1A"
    primaryBorderColor: "#D97757"
    lineColor: "#8B5E3C"
    secondaryColor: "#F5F1E8"
    tertiaryColor: "#FAF9F5"
    fontFamily: "ui-sans-serif, system-ui, -apple-system, sans-serif"
    fontSize: "14px"
---
flowchart LR
    A[query + engine] --> B{engine=auto?}
    B -- Yes --> C[Detect language]
    C --> D[Build engine plan]
    B -- No --> E[Use specified engine]
    D --> F[Try engines in order]
    E --> F
    F --> G{Results?}
    G -- Yes --> H[Aggregate + deduplicate]
    G -- No, next engine --> F
    H --> I[Return results]

Installation

Quick install (recommended)

Step 1: Create a virtual environment

# macOS/Linux - using system Python 3 (3.10-3.13)
python3 -m venv crawl4ai
source crawl4ai/bin/activate

# Windows
python -m venv crawl4ai
crawl4ai\Scripts\activate

Step 2: Install

pip install --upgrade pip
pip install crawl4agent
playwright install chromium

Alternative methods

If python3 is too old (3.9 or below):

# Use specific Python version (3.10, 3.11, 3.12, or 3.13)
python3.12 -m venv crawl4ai
source crawl4ai/bin/activate
pip install crawl4agent

Using conda:

conda create -n crawl4ai python=3.12
conda activate crawl4ai
pip install crawl4agent
playwright install chromium

Using pipx (global command):

pipx install crawl4agent
crawl4ai-mcp --help

Troubleshooting

Problem: "pip install" uses Python 2.7

# macOS: use python3 explicitly
python3 -m pip install crawl4agent

# Or check which pip you're using
which pip
pip --version

Problem: "No matching distribution found for crawl4agent"

• Check Python version: python3 --version (must be 3.10-3.13)

• Upgrade pip: python3 -m pip install --upgrade pip

Problem: "playwright install" fails

• Use mirror (China): export PLAYWRIGHT_DOWNLOAD_HOST=https://npmmirror.com/mirrors/playwright/

• Then: python3 -m playwright install chromium

Why this project exists

Most generic “web fetch” tools either fail on JS-heavy pages or return too much boilerplate. This project focuses on four things:

• Non-LLM quality first: usable even with zero model config

• Minimal MCP surface: easier for agents, easier to maintain

• Pragmatic anti-bot workflow: proxy / cookies / persistent profile are first-class

• Golden regression review: full markdown outputs can be saved and inspected page by page

Core capabilities

Non-LLM mode

Optional LLM mode

Important reality check:

• With llm_instruction, the prompt is constraint-heavy and biased toward preserving original lines.

• Without llm_instruction, the model does a more generic “clean readable markdown” pass.

• If the LLM call fails for any reason, the tool returns the original non-LLM extraction plus llm_used=false and llm_error.

MCP Tools

fetch_urls

{
  "urls": ["https://a.com", "https://b.com"],
  "format": "markdown",
  "max_chars": 200000,
  "concurrency": 3,
  "use_llm": false,
  "llm_instruction": "keep only the tutorial body and in-body references"
}

Use a single-element list if you only need one page.

For supported video URLs (youtube.com, youtu.be, bilibili.com, b23.tv), fetch_urls prefers transcript extraction and returns readable markdown built from subtitles when available.

Return shape

search_web

{
  "query": "crawl4ai web scraping",
  "engine": "auto",
  "max_results": 10,
  "lang": ""
}

When engine="auto", the server tries engines in fallback order: DuckDuckGo → Bing → Google → Baidu. The first engine that returns results wins.

Search return shape

Anti-bot realism

The server already includes randomized browser behavior in code:

Note: For overseas websites (Medium, ProductHunt, etc.), using a proxy is recommended. The server supports HTTP/HTTPS/SOCKS5 proxies via CRAWL4AI_MCP_PROXY environment variable.

Proxy input formats

CRAWL4AI_MCP_PROXY accepts all of these:

That means the README should not claim “perfect stealth”, but it can honestly claim human-like randomization and practical anti-bot knobs.

Quickstart

Conda

conda env create -f environment.yml
conda activate crawl4ai-mcp
python -m playwright install
crawl4ai-mcp

venv

python -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
python -m pip install -e '.[dev]'
python -m playwright install
crawl4ai-mcp

MCP server config example

{
  "mcpServers": {
    "crawl4ai": {
      "command": "crawl4ai-mcp",
      "env": {
        "CRAWL4AI_MCP_HEADLESS": "true",
        "CRAWL4AI_MCP_PROXY": "127.0.0.1:7890",
        "CRAWL4AI_MCP_NAVIGATION_TIMEOUT_MS": "30000",
        "CRAWL4AI_MCP_WAIT_UNTIL": "load",

        "OPENAI_BASE_URL": "https://your-openai-compatible-host",
        "OPENAI_API_KEY": "your-api-key",
        "OPENAI_MODEL": "your-model-name"
      }
    }
  }
}

LLM-related env vars are optional. use_llm is still off by default at call time. If any LLM env is missing, invalid, or the model call fails, the server automatically falls back to non-LLM extraction.

Runtime configuration

One-shot CLI

This project now exposes a stateless one-shot CLI in addition to the MCP stdio server.

Fetch a single URL once and print JSON:

crawl4agent fetch "https://obsidian.md/help/cli" --format markdown

Search the web once and print JSON:

crawl4agent search "agent framework" --engine auto --max-results 5

Use proxy and browser cookies for video transcript extraction:

crawl4agent fetch "https://www.youtube.com/watch?v=OFfwN23hR8U" \
  --proxy http://127.0.0.1:7890 \
  --cookies-from-browser chrome

Run golden smoke once and print a JSON array:

crawl4agent smoke --out-dir ./_golden_outputs

The existing crawl4ai-mcp command remains the MCP stdio server entrypoint for MCP hosts.

Available help surfaces:

crawl4agent --help
crawl4agent fetch --help
crawl4agent search --help
crawl4agent smoke --help

Golden smoke regression

CRAWL4AI_MCP_SMOKE_DIR=./_golden_outputs .venv/bin/python -m crawl4ai_mcp.smoke_golden

For overseas video URLs, a local proxy is often needed:

CRAWL4AI_MCP_PROXY=http://127.0.0.1:7890 \
CRAWL4AI_MCP_SMOKE_DIR=./_golden_outputs \
.venv/bin/python -m crawl4ai_mcp.smoke_golden

This writes full markdown outputs to _golden_outputs/ so you can inspect extraction quality page by page.

The golden set now includes the earlier baseline URLs plus ainew.me, openclaw, watcha, producthunt, mydrivers, caihongtu, openrouter, mobile Douban, and video pages from YouTube / Bilibili. For sites outside mainland China, proxy-based verification is recommended.

Some overseas sites may still return Cloudflare or similar verification pages even when a proxy is configured. In those cases the server now marks them with blocked=true. The recommended path is: better proxy quality, valid cookies, or a persistent browser profile after manual verification.

For some video golden URLs, subtitle extraction may require login. If yt-dlp reports login-required subtitles, configure either CRAWL4AI_MCP_YTDLP_COOKIES_FROM_BROWSER or CRAWL4AI_MCP_YTDLP_COOKIEFILE before running golden smoke.

Prior art

• Crawl4AI: https://github.com/unclecode/crawl4ai

• mcp-crawl4ai-rag: https://github.com/coleam00/mcp-crawl4ai-rag

• weidwonder/crawl4ai-mcp-server: https://github.com/weidwonder/crawl4ai-mcp-server

• WaterCrawl: https://github.com/watercrawl/WaterCrawl

• teracrawl: https://github.com/BrowserCash/teracrawl

License

This project is licensed under AGPL-3.0-or-later.