Which integrations are available for this server?

Provides tools to crawl websites using Cloudflare's Browser Rendering Crawl API, with support for authentication, custom headers, AI extraction, and job management. Leverages Cloudflare Workers AI for structured data extraction during crawling (requires 'json' format).

How do I use mcp-cloudflare-crawl?

1. Click on "Deploy Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@mcp-cloudflare-crawl crawl https://example.com and return markdown content" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

mcp-cloudflare-crawl

by shihaku1223

Overview Schema Source Related Servers Discussions Admin

Python

Remote

mcp-cloudflare-crawl

An MCP server that exposes Cloudflare's Browser Rendering Crawl API as tools for LLM clients.

Requirements

uv
A Cloudflare account with Browser Rendering enabled
A Cloudflare API token with Browser Rendering - Edit permission

Related MCP server: ScrapeGraph MCP Server

Setup

cp .env.example .env
# Edit .env and fill in your credentials

.env:

CLOUDFLARE_API_TOKEN=your_api_token_here
CLOUDFLARE_ACCOUNT_ID=your_account_id_here

Running

stdio (default — for Claude Desktop and most MCP clients)

uv run mcp-cloudflare-crawl

Streamable HTTP

uv run mcp-cloudflare-crawl --transport streamable-http
# Listens on http://127.0.0.1:8000/mcp by default

uv run mcp-cloudflare-crawl --transport streamable-http --host 0.0.0.0 --port 9000

Claude Code Integration

claude mcp add \
  --env CLOUDFLARE_API_TOKEN=your_api_token_here \
  --env CLOUDFLARE_ACCOUNT_ID=your_account_id_here \
  cloudflare-crawl \
  -- uv run --directory /absolute/path/to/mcp-cloudflare-crawl mcp-cloudflare-crawl

Add --scope user to make it available across all projects:

claude mcp add --scope user \
  --env CLOUDFLARE_API_TOKEN=your_api_token_here \
  --env CLOUDFLARE_ACCOUNT_ID=your_account_id_here \
  cloudflare-crawl \
  -- uv run --directory /absolute/path/to/mcp-cloudflare-crawl mcp-cloudflare-crawl

Claude Desktop Integration

Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "cloudflare-crawl": {
      "command": "uv",
      "args": [
        "run",
        "--directory",
        "/absolute/path/to/mcp-cloudflare-crawl",
        "mcp-cloudflare-crawl"
      ],
      "env": {
        "CLOUDFLARE_API_TOKEN": "your_api_token_here",
        "CLOUDFLARE_ACCOUNT_ID": "your_account_id_here"
      }
    }
  }
}

Testing with curl

The server uses SSE (Server-Sent Events) format. Responses look like:

event: message
data: {"jsonrpc":"2.0","id":1,"result":{...}}

To parse with jq, extract the data: line first:

curl -s -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "mcp-session-id: $SESSION_ID" \
  -d '{...}' \
  | grep '^data:' | sed 's/^data: //' | jq .

Step 1 — Initialize session and capture session ID

SESSION_ID=$(curl -s -D - -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{
    "jsonrpc": "2.0", "id": 1, "method": "initialize",
    "params": {
      "protocolVersion": "2024-11-05",
      "capabilities": {},
      "clientInfo": {"name": "curl-test", "version": "1.0"}
    }
  }' | grep -i '^mcp-session-id:' | awk '{print $2}' | tr -d '\r')

echo "Session ID: $SESSION_ID"

Step 2 — List available tools

curl -s -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "mcp-session-id: $SESSION_ID" \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}' \
  | grep '^data:' | sed 's/^data: //' | jq .

Step 3 — Start a crawl (all optional parameters)

curl -s -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "mcp-session-id: $SESSION_ID" \
  -d '{
    "jsonrpc": "2.0", "id": 3, "method": "tools/call",
    "params": {
      "name": "crawl_start",
      "arguments": {
        "url": "https://www.exampledocs.com/docs/",
        "crawl_purposes": ["search"],
        "limit": 50,
        "depth": 2,
        "formats": ["markdown"],
        "render": true,
        "max_age": 7200,
        "source": "all",
        "include_external_links": true,
        "include_subdomains": true,
        "include_patterns": ["**/api/v1/*"],
        "exclude_patterns": ["*/learning-paths/*"],
        "reject_resource_types": ["image", "media", "font"],
        "goto_options": {"waitUntil": "networkidle2", "timeout": 30000},
        "wait_for_selector": {"selector": "#content", "timeout": 5000}
      }
    }
  }' | grep '^data:' | sed 's/^data: //' | jq .

Step 4 — Poll status

curl -s -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "mcp-session-id: $SESSION_ID" \
  -d '{
    "jsonrpc": "2.0", "id": 4, "method": "tools/call",
    "params": {
      "name": "crawl_status",
      "arguments": {"job_id": "YOUR_JOB_ID"}
    }
  }' | grep '^data:' | sed 's/^data: //' | jq .

Step 5 — Crawl with AI structured extraction

Requires "json" in formats. Uses Cloudflare Workers AI and incurs additional charges.

curl -s -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "mcp-session-id: $SESSION_ID" \
  -d '{
    "jsonrpc": "2.0", "id": 5, "method": "tools/call",
    "params": {
      "name": "crawl_and_wait",
      "arguments": {
        "url": "https://example.com/",
        "formats": ["json"],
        "limit": 5,
        "json_options": {
          "prompt": "Extract product names and prices",
          "response_format": {"type": "object"}
        },
        "timeout": 120.0
      }
    }
  }' | grep '^data:' | sed 's/^data: //' | jq .

Step 6 — Crawl a password-protected site

curl -s -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "mcp-session-id: $SESSION_ID" \
  -d '{
    "jsonrpc": "2.0", "id": 6, "method": "tools/call",
    "params": {
      "name": "crawl_start",
      "arguments": {
        "url": "https://internal.example.com/docs/",
        "authenticate": {"username": "user", "password": "pass"},
        "extra_http_headers": {"X-API-Key": "abc123"},
        "cookies": [{"name": "session", "value": "xyz", "domain": "internal.example.com"}],
        "formats": ["markdown"]
      }
    }
  }' | grep '^data:' | sed 's/^data: //' | jq .

Step 7 — List all stored jobs

curl -s -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "mcp-session-id: $SESSION_ID" \
  -d '{
    "jsonrpc": "2.0", "id": 7, "method": "tools/call",
    "params": {
      "name": "crawl_list",
      "arguments": {}
    }
  }' | grep '^data:' | sed 's/^data: //' | jq .

Tools

`crawl_start`

Submit a crawl job. Returns a job_id immediately — crawling happens asynchronously. The job is automatically saved to the local SQLite database.

Parameter	Type	Description
`url`	string	Required. Starting URL to crawl
`limit`	int	Max pages to crawl (default: 10, max: 100,000)
`depth`	int	Max link depth (default: 100,000)
`source`	string	URL discovery: `"all"`, `"sitemaps"`, or `"links"`
`formats`	list[string]	Output formats: `"html"`, `"markdown"`, `"json"`
`render`	bool	Execute JavaScript via headless browser (default: true)
`max_age`	int	Cache duration in seconds (default: 86400, max: 604800)
`modified_since`	int	Unix timestamp — only crawl pages modified since then
`crawl_purposes`	list[string]	Declare use: `"search"`, `"ai-input"`, `"ai-train"`
`include_patterns`	list[string]	URL patterns to include (`` = no slash, `*` = any)
`exclude_patterns`	list[string]	URL patterns to exclude (higher priority than include)
`include_external_links`	bool	Follow links to external domains
`include_subdomains`	bool	Follow links to subdomains
`authenticate`	dict	HTTP auth credentials: `{"username": "...", "password": "..."}`
`extra_http_headers`	dict	Custom request headers: `{"X-API-Key": "..."}`
`json_options`	dict	AI extraction config (requires `"json"` in formats). Keys: `"prompt"`, `"response_format"`, `"custom_ai"`
`cookies`	list[dict]	Browser cookies: `[{"name": "...", "value": "...", "domain": "..."}]`
`goto_options`	dict	Navigation behaviour: `{"waitUntil": "networkidle2", "timeout": 30000}`
`wait_for_selector`	dict	Wait for DOM element: `{"selector": "#content", "timeout": 5000, "visible": true}`
`reject_resource_types`	list[string]	Block resource types: `"image"`, `"media"`, `"font"`, `"stylesheet"`, `"script"`

Response:

{ "job_id": "c7f8s2d9-a8e7-4b6e-8e4d-3d4a1b2c3f4e" }

`crawl_status`

Poll the status and results of a crawl job. Also updates the job's status in the local database.

Parameter	Type	Description
`job_id`	string	Required. Job ID from `crawl_start`
`cursor`	int	Pagination token for large result sets (>10 MB)
`limit`	int	Records per page
`status_filter`	string	Filter by record status: `queued`, `completed`, `disallowed`, `skipped`, `errored`, `cancelled`

Response:

{
  "id": "c7f8s2d9-...",
  "status": "completed",
  "total": 20,
  "finished": 20,
  "browser_seconds_used": 134.7,
  "cursor": null,
  "records": [
    {
      "url": "https://example.com/",
      "status": "completed",
      "markdown": "# Example Domain\n...",
      "metadata": { "status": 200, "title": "Example Domain", "url": "https://example.com/" }
    }
  ]
}

Job statuses: running · completed · errored · cancelled_due_to_timeout · cancelled_due_to_limits · cancelled_by_user

Record statuses: queued · completed · errored · disallowed · skipped · cancelled

`crawl_cancel`

Cancel a running crawl job.

Parameter	Type	Description
`job_id`	string	Required. Job ID from `crawl_start`

`crawl_and_wait`

Start a crawl and block until it completes, returning the final results. Combines crawl_start and crawl_status polling in one call. The job is saved and status is updated in the local database throughout.

Accepts all parameters from crawl_start, plus:

Parameter	Type	Description
`poll_interval`	float	Seconds between status polls (default: 5.0)
`timeout`	float	Max seconds to wait (default: 300.0)

Use this for small crawls (a few pages). For large crawls, use crawl_start + crawl_status separately to avoid timeouts.

`crawl_list`

List all crawl jobs stored in the local SQLite database. Jobs are recorded automatically on crawl_start and crawl_and_wait, and their status is updated on every crawl_status or crawl_cancel call.

Parameter	Type	Description
`status_filter`	string	Filter by job status (see below)
`limit`	int	Max jobs to return (default: 50)
`offset`	int	Jobs to skip for pagination (default: 0)

Job statuses: submitted · running · completed · errored · cancelled_due_to_timeout · cancelled_due_to_limits · cancelled_by_user

Response:

{
  "jobs": [
    {
      "job_id": "c7f8s2d9-...",
      "url": "https://example.com/",
      "status": "completed",
      "created_at": "2026-03-25T00:00:00+00:00",
      "updated_at": "2026-03-25T00:01:00+00:00"
    }
  ],
  "count": 1
}

Job Database

Jobs are persisted in a local SQLite database across server restarts.

Default location: ~/.local/share/mcp-cloudflare-crawl/jobs.db

Override with environment variable:

MCP_DB_PATH=/path/to/custom/jobs.db

Development

# Install dependencies
uv sync

# Run tests
uv run pytest

# Run tests with verbose output
uv run pytest -v

Notes

The Cloudflare Crawl API is asynchronous — crawl_start returns immediately, results are retrieved via crawl_status.
The crawler respects robots.txt by default. Disallowed URLs appear with "status": "disallowed".
The json format uses Workers AI for structured extraction and incurs additional charges.
Setting render: false skips the headless browser and fetches static HTML — faster and currently unbilled during beta.
Results are retained for 14 days after a job completes. Maximum job runtime is 7 days.
The crawler identifies itself as CloudflareBrowserRenderingCrawler/1.0 and cannot bypass Cloudflare protection or CAPTCHAs.
HTTP 429 (rate limit) responses are automatically retried with exponential backoff (up to 3 retries: 1s → 2s → 4s). The Retry-After response header is respected when present.

License

MIT

Available Tools

5 tools

crawl_and_waitA

Start a crawl and wait for it to complete, returning the final results.

This is a convenience tool that combines crawl_start and crawl_status polling. Suitable for small crawls (few pages). For large crawls, use crawl_start and crawl_status separately to avoid timeout issues.

Args: url: The starting URL to crawl (required). limit: Maximum pages to crawl (default: 10, max: 100000). depth: Maximum link depth (default: 100000). source: URL discovery source — "all", "sitemaps", or "links" (default: "all"). formats: Output formats — any of ["html", "markdown", "json"] (default: ["html"]). Note: "json" uses Workers AI and incurs additional charges. render: Whether to execute JavaScript via headless browser (default: true). max_age: Cache duration in seconds (default: 86400, max: 604800). modified_since: Unix timestamp — only crawl pages modified since this time. crawl_purposes: Declare content use — any of ["search", "ai-input", "ai-train"]. include_patterns: URL patterns to include (* = any chars except /, ** = any chars). exclude_patterns: URL patterns to exclude (takes priority over include_patterns). include_external_links: Whether to follow links to external domains. include_subdomains: Whether to follow links to subdomains. authenticate: HTTP authentication credentials for protected sites. Example: {"username": "user", "password": "pass"}. extra_http_headers: Custom HTTP headers to send with each crawl request. Example: {"X-API-Key": "abc123"}. json_options: AI-based structured data extraction config (requires "json" in formats). Keys: "prompt" (str) — extraction instruction, "response_format" (dict) — JSON schema for output, "custom_ai" (dict) — custom AI model config. cookies: Browser cookies to set during the crawl. Example: [{"name": "session", "value": "abc", "domain": "example.com"}]. goto_options: Page navigation behaviour. Keys: "waitUntil" (str) — e.g. "networkidle2", "load", "domcontentloaded"; "timeout" (int) — navigation timeout in milliseconds. wait_for_selector: Wait for a DOM element before scraping each page. Keys: "selector" (str), "timeout" (int, ms), "visible" (bool). reject_resource_types: Resource types to block to speed up crawls and reduce cost. Values: "image", "media", "font", "stylesheet", "script", etc. poll_interval: Seconds between status polls (default: 5.0). timeout: Maximum seconds to wait for completion (default: 300.0).

Returns: Final crawl result (same shape as crawl_status) once the job completes, or raises RuntimeError if the timeout is exceeded.

Inputs: url (required), depth, limit, render, +18 more. Returns: object

crawl_cancelB

Cancel a running crawl job.

Args: job_id: The crawl job ID returned by crawl_start.

Returns: {"success": true, "job_id": "<job_id>"}

Inputs: job_id (required). Returns: object

crawl_listA

List all crawl jobs stored in the local database.

Jobs are recorded automatically when crawl_start or crawl_and_wait is called. Status is updated whenever crawl_status is polled.

Args: status_filter: Filter by job status — one of: "submitted", "running", "completed", "errored", "cancelled_due_to_timeout", "cancelled_due_to_limits", "cancelled_by_user". limit: Maximum number of jobs to return (default: 50). offset: Number of jobs to skip for pagination (default: 0).

Returns: { "jobs": [ { "job_id": "...", "url": "https://...", "status": "completed", "created_at": "2026-03-25T00:00:00+00:00", "updated_at": "2026-03-25T00:01:00+00:00" }, ... ], "count": }

Inputs: limit, offset, status_filter. Returns: object

crawl_startA

Start an asynchronous crawl job using Cloudflare's Browser Rendering Crawl API.

Submits a crawl job and returns a job_id immediately. Use crawl_status to poll for results, or use crawl_and_wait to block until completion.

Args: url: The starting URL to crawl (required). limit: Maximum number of pages to crawl (default: 10, max: 100000). depth: Maximum link depth to follow (default: 100000). source: URL discovery source — "all", "sitemaps", or "links" (default: "all"). formats: Output formats — any of ["html", "markdown", "json"] (default: ["html"]). Note: "json" uses Workers AI and incurs additional charges. render: Whether to execute JavaScript via headless browser (default: true). Set false for faster, unbilled static HTML fetching. max_age: Cache duration in seconds (default: 86400, max: 604800). modified_since: Unix timestamp — only crawl pages modified since this time. crawl_purposes: Declare content use — any of ["search", "ai-input", "ai-train"]. include_patterns: URL patterns to include (* = any chars except /, ** = any chars). exclude_patterns: URL patterns to exclude (takes priority over include_patterns). include_external_links: Whether to follow links to external domains. include_subdomains: Whether to follow links to subdomains. authenticate: HTTP authentication credentials for protected sites. Example: {"username": "user", "password": "pass"}. extra_http_headers: Custom HTTP headers to send with each crawl request. Example: {"X-API-Key": "abc123"}. json_options: AI-based structured data extraction config (requires "json" in formats). Keys: "prompt" (str) — extraction instruction, "response_format" (dict) — JSON schema for output, "custom_ai" (dict) — custom AI model config. cookies: Browser cookies to set during the crawl. Example: [{"name": "session", "value": "abc", "domain": "example.com"}]. goto_options: Page navigation behaviour. Keys: "waitUntil" (str) — e.g. "networkidle2", "load", "domcontentloaded"; "timeout" (int) — navigation timeout in milliseconds. wait_for_selector: Wait for a DOM element before scraping each page. Keys: "selector" (str), "timeout" (int, ms), "visible" (bool). reject_resource_types: Resource types to block to speed up crawls and reduce cost. Values: "image", "media", "font", "stylesheet", "script", etc.

Returns: {"job_id": ""} — use this ID with crawl_status or crawl_cancel.

Inputs: url (required), depth, limit, render, +16 more. Returns: object

crawl_statusA

Check the status and retrieve results of a crawl job.

For large result sets (>10 MB), the response includes a "cursor" value. Pass it back in the next call to paginate through results.

Args: job_id: The crawl job ID returned by crawl_start. cursor: Pagination token from a previous response (for large result sets). limit: Number of records to return per page. status_filter: Filter records by status — one of: "queued", "completed", "disallowed", "skipped", "errored", "cancelled".

Returns: { "id": "<job_id>", "status": "running|completed|errored|cancelled_due_to_timeout|cancelled_due_to_limits|cancelled_by_user", "total": , "finished": , "browser_seconds_used": , "cursor": <int|null>, "records": [ { "url": "...", "status": "completed|errored|queued|disallowed|skipped|cancelled", "html": "...", # if html format requested "markdown": "...", # if markdown format requested "metadata": {"status": 200, "title": "...", "url": "..."} }, ... ] }

Inputs: job_id (required), limit, cursor, status_filter. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

Deploy Server

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A4/5.0

Scored 2026-09-13 13:00 across 5 tools

Disambiguation4/5

crawl_start, crawl_status, crawl_cancel, and crawl_list each have distinct purposes. crawl_and_wait overlaps with crawl_start+crawl_status polling, but its description clearly frames it as a convenience wrapper for small crawls, so confusion risk is limited.

Naming Consistency5/5

All tools follow the same crawl_ + verb/short-noun pattern (crawl_start, crawl_status, crawl_cancel, crawl_list, crawl_and_wait) in consistent snake_case.

Tool Count5/5

Five tools is well-scoped for an asynchronous crawl API, covering submission, polling, cancellation, listing, and a blocking convenience path without redundancy bloat.

Completeness4/5

Full job lifecycle is covered (start, status, cancel, list, and synchronous wait). Only minor gaps exist, such as deleting/clearing stored jobs or re-fetching a single job by ID.

Maintenance

ActivityInactive

ResponsivenessNo recent community issues

Resources

Related MCP Connectors

Geonode MCP ServerA
com.geonode
Web data for agents. Extract web pages, crawl websites, discover URLs, and search the web.
Web Scraping Browser Automation Search
161
https://mcp.geonode.com/mcp
AdsCrawl
io.github.AdsCrawl
Cloud browser, web extraction, screenshots, and remote CDP tools for AI agents.
Browser Automation Web Scraping Agent Orchestration
1
https://api.adscrawl.net/mcp
webclaw
io.github.0xMassi
Web scraping, search, crawling, structured extraction and research for AI agents.
Web Scraping Search Research & Data
12,436
https://api.webclaw.io/mcp
Fastcrawl
net.fastcrawl
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
Browser Automation Web Scraping
1
https://mcp.fastcrawl.net/mcp

Related MCP Servers

Hyperbrowser MCP Server
Web Scraping Browser Automation Agent Orchestration
BACH-AI-Tools
A
license
Not graded
quality
Not graded
maintenance
Provides tools to scrape, extract structured data, and crawl webpages, with access to browser automation agents like OpenAI's CUA, Anthropic's Claude Computer Use, and Browser Use for complex web tasks.
Updated a year ago (2025-11-19 03:37 UTC)
-
ScrapeGraph MCP Server
Web Scraping Browser Automation Search
SungchanHussein
F
license
Not graded
quality
A
maintenance
Enables language models to scrape and extract web content, run searches, crawl multiple pages, generate JSON schemas, schedule monitors, and manage account history through a unified protocol interface.
Updated a month ago (2026-10-16 10:05 UTC)
-
FlyCrawl MCP Serverofficial
Browser Automation Web Scraping Search
FlyCrawl-ai
A
license
A
quality
C
maintenance
Enables Claude Desktop, Cursor AI, Windsurf, and autonomous LLM agents to scrape, crawl, map, search, and extract validated JSON from live web pages with dynamic rendering, anti-bot bypass, and Markdown conversion.
Updated a month ago (2026-10-17 13:08 UTC)
5
11 npm
MIT
ScrapeGraph MCP Server
Web Scraping Browser Automation Search
hansioannidis30
F
license
Not graded
quality
A
maintenance
Enables language models to perform AI-powered web scraping, structured data extraction, web search, crawling, schema generation, and monitor management through integrated API tools.
Updated 24 days ago (2026-10-18 19:25 UTC)
-

mcp-cloudflare-crawl

Requirements

Setup

Running

stdio (default — for Claude Desktop and most MCP clients)

Streamable HTTP

Claude Code Integration

Claude Desktop Integration

Testing with curl

Step 1 — Initialize session and capture session ID

Step 2 — List available tools

Step 3 — Start a crawl (all optional parameters)

Step 4 — Poll status

Step 5 — Crawl with AI structured extraction

Step 6 — Crawl a password-protected site

Step 7 — List all stored jobs

Tools

crawl_start

crawl_status

crawl_cancel

crawl_and_wait

crawl_list

Job Database

Development

Notes

License

Available Tools

Tool Schema Changelog

TDQS

Maintenance

Resources

Related MCP Connectors

Related MCP Servers

Hyperbrowser MCP Server

ScrapeGraph MCP Server

FlyCrawl MCP Serverofficial

ScrapeGraph MCP Server

`crawl_start`

`crawl_status`

`crawl_cancel`

`crawl_and_wait`

`crawl_list`