Skip to main content
Glama
shihaku1223

mcp-cloudflare-crawl

by shihaku1223

mcp-cloudflare-crawl

An MCP server that exposes Cloudflare's Browser Rendering Crawl API as tools for LLM clients.

Requirements

  • uv

  • A Cloudflare account with Browser Rendering enabled

  • A Cloudflare API token with Browser Rendering - Edit permission

Related MCP server: Hyperbrowser MCP Server

Setup

cp .env.example .env
# Edit .env and fill in your credentials

.env:

CLOUDFLARE_API_TOKEN=your_api_token_here
CLOUDFLARE_ACCOUNT_ID=your_account_id_here

Running

stdio (default — for Claude Desktop and most MCP clients)

uv run mcp-cloudflare-crawl

Streamable HTTP

uv run mcp-cloudflare-crawl --transport streamable-http
# Listens on http://127.0.0.1:8000/mcp by default

uv run mcp-cloudflare-crawl --transport streamable-http --host 0.0.0.0 --port 9000

Claude Code Integration

claude mcp add \
  --env CLOUDFLARE_API_TOKEN=your_api_token_here \
  --env CLOUDFLARE_ACCOUNT_ID=your_account_id_here \
  cloudflare-crawl \
  -- uv run --directory /absolute/path/to/mcp-cloudflare-crawl mcp-cloudflare-crawl

Add --scope user to make it available across all projects:

claude mcp add --scope user \
  --env CLOUDFLARE_API_TOKEN=your_api_token_here \
  --env CLOUDFLARE_ACCOUNT_ID=your_account_id_here \
  cloudflare-crawl \
  -- uv run --directory /absolute/path/to/mcp-cloudflare-crawl mcp-cloudflare-crawl

Claude Desktop Integration

Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "cloudflare-crawl": {
      "command": "uv",
      "args": [
        "run",
        "--directory",
        "/absolute/path/to/mcp-cloudflare-crawl",
        "mcp-cloudflare-crawl"
      ],
      "env": {
        "CLOUDFLARE_API_TOKEN": "your_api_token_here",
        "CLOUDFLARE_ACCOUNT_ID": "your_account_id_here"
      }
    }
  }
}

Testing with curl

The server uses SSE (Server-Sent Events) format. Responses look like:

event: message
data: {"jsonrpc":"2.0","id":1,"result":{...}}

To parse with jq, extract the data: line first:

curl -s -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "mcp-session-id: $SESSION_ID" \
  -d '{...}' \
  | grep '^data:' | sed 's/^data: //' | jq .

Step 1 — Initialize session and capture session ID

SESSION_ID=$(curl -s -D - -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{
    "jsonrpc": "2.0", "id": 1, "method": "initialize",
    "params": {
      "protocolVersion": "2024-11-05",
      "capabilities": {},
      "clientInfo": {"name": "curl-test", "version": "1.0"}
    }
  }' | grep -i '^mcp-session-id:' | awk '{print $2}' | tr -d '\r')

echo "Session ID: $SESSION_ID"

Step 2 — List available tools

curl -s -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "mcp-session-id: $SESSION_ID" \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}' \
  | grep '^data:' | sed 's/^data: //' | jq .

Step 3 — Start a crawl (all optional parameters)

curl -s -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "mcp-session-id: $SESSION_ID" \
  -d '{
    "jsonrpc": "2.0", "id": 3, "method": "tools/call",
    "params": {
      "name": "crawl_start",
      "arguments": {
        "url": "https://www.exampledocs.com/docs/",
        "crawl_purposes": ["search"],
        "limit": 50,
        "depth": 2,
        "formats": ["markdown"],
        "render": true,
        "max_age": 7200,
        "source": "all",
        "include_external_links": true,
        "include_subdomains": true,
        "include_patterns": ["**/api/v1/*"],
        "exclude_patterns": ["*/learning-paths/*"],
        "reject_resource_types": ["image", "media", "font"],
        "goto_options": {"waitUntil": "networkidle2", "timeout": 30000},
        "wait_for_selector": {"selector": "#content", "timeout": 5000}
      }
    }
  }' | grep '^data:' | sed 's/^data: //' | jq .

Step 4 — Poll status

curl -s -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "mcp-session-id: $SESSION_ID" \
  -d '{
    "jsonrpc": "2.0", "id": 4, "method": "tools/call",
    "params": {
      "name": "crawl_status",
      "arguments": {"job_id": "YOUR_JOB_ID"}
    }
  }' | grep '^data:' | sed 's/^data: //' | jq .

Step 5 — Crawl with AI structured extraction

Requires "json" in formats. Uses Cloudflare Workers AI and incurs additional charges.

curl -s -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "mcp-session-id: $SESSION_ID" \
  -d '{
    "jsonrpc": "2.0", "id": 5, "method": "tools/call",
    "params": {
      "name": "crawl_and_wait",
      "arguments": {
        "url": "https://example.com/",
        "formats": ["json"],
        "limit": 5,
        "json_options": {
          "prompt": "Extract product names and prices",
          "response_format": {"type": "object"}
        },
        "timeout": 120.0
      }
    }
  }' | grep '^data:' | sed 's/^data: //' | jq .

Step 6 — Crawl a password-protected site

curl -s -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "mcp-session-id: $SESSION_ID" \
  -d '{
    "jsonrpc": "2.0", "id": 6, "method": "tools/call",
    "params": {
      "name": "crawl_start",
      "arguments": {
        "url": "https://internal.example.com/docs/",
        "authenticate": {"username": "user", "password": "pass"},
        "extra_http_headers": {"X-API-Key": "abc123"},
        "cookies": [{"name": "session", "value": "xyz", "domain": "internal.example.com"}],
        "formats": ["markdown"]
      }
    }
  }' | grep '^data:' | sed 's/^data: //' | jq .

Step 7 — List all stored jobs

curl -s -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "mcp-session-id: $SESSION_ID" \
  -d '{
    "jsonrpc": "2.0", "id": 7, "method": "tools/call",
    "params": {
      "name": "crawl_list",
      "arguments": {}
    }
  }' | grep '^data:' | sed 's/^data: //' | jq .

Tools

crawl_start

Submit a crawl job. Returns a job_id immediately — crawling happens asynchronously. The job is automatically saved to the local SQLite database.

Parameter

Type

Description

url

string

Required. Starting URL to crawl

limit

int

Max pages to crawl (default: 10, max: 100,000)

depth

int

Max link depth (default: 100,000)

source

string

URL discovery: "all", "sitemaps", or "links"

formats

list[string]

Output formats: "html", "markdown", "json"

render

bool

Execute JavaScript via headless browser (default: true)

max_age

int

Cache duration in seconds (default: 86400, max: 604800)

modified_since

int

Unix timestamp — only crawl pages modified since then

crawl_purposes

list[string]

Declare use: "search", "ai-input", "ai-train"

include_patterns

list[string]

URL patterns to include (* = no slash, ** = any)

exclude_patterns

list[string]

URL patterns to exclude (higher priority than include)

include_external_links

bool

Follow links to external domains

include_subdomains

bool

Follow links to subdomains

authenticate

dict

HTTP auth credentials: {"username": "...", "password": "..."}

extra_http_headers

dict

Custom request headers: {"X-API-Key": "..."}

json_options

dict

AI extraction config (requires "json" in formats). Keys: "prompt", "response_format", "custom_ai"

cookies

list[dict]

Browser cookies: [{"name": "...", "value": "...", "domain": "..."}]

goto_options

dict

Navigation behaviour: {"waitUntil": "networkidle2", "timeout": 30000}

wait_for_selector

dict

Wait for DOM element: {"selector": "#content", "timeout": 5000, "visible": true}

reject_resource_types

list[string]

Block resource types: "image", "media", "font", "stylesheet", "script"

Response:

{ "job_id": "c7f8s2d9-a8e7-4b6e-8e4d-3d4a1b2c3f4e" }

crawl_status

Poll the status and results of a crawl job. Also updates the job's status in the local database.

Parameter

Type

Description

job_id

string

Required. Job ID from crawl_start

cursor

int

Pagination token for large result sets (>10 MB)

limit

int

Records per page

status_filter

string

Filter by record status: queued, completed, disallowed, skipped, errored, cancelled

Response:

{
  "id": "c7f8s2d9-...",
  "status": "completed",
  "total": 20,
  "finished": 20,
  "browser_seconds_used": 134.7,
  "cursor": null,
  "records": [
    {
      "url": "https://example.com/",
      "status": "completed",
      "markdown": "# Example Domain\n...",
      "metadata": { "status": 200, "title": "Example Domain", "url": "https://example.com/" }
    }
  ]
}

Job statuses: running · completed · errored · cancelled_due_to_timeout · cancelled_due_to_limits · cancelled_by_user

Record statuses: queued · completed · errored · disallowed · skipped · cancelled


crawl_cancel

Cancel a running crawl job.

Parameter

Type

Description

job_id

string

Required. Job ID from crawl_start


crawl_and_wait

Start a crawl and block until it completes, returning the final results. Combines crawl_start and crawl_status polling in one call. The job is saved and status is updated in the local database throughout.

Accepts all parameters from crawl_start, plus:

Parameter

Type

Description

poll_interval

float

Seconds between status polls (default: 5.0)

timeout

float

Max seconds to wait (default: 300.0)

Use this for small crawls (a few pages). For large crawls, use crawl_start + crawl_status separately to avoid timeouts.


crawl_list

List all crawl jobs stored in the local SQLite database. Jobs are recorded automatically on crawl_start and crawl_and_wait, and their status is updated on every crawl_status or crawl_cancel call.

Parameter

Type

Description

status_filter

string

Filter by job status (see below)

limit

int

Max jobs to return (default: 50)

offset

int

Jobs to skip for pagination (default: 0)

Job statuses: submitted · running · completed · errored · cancelled_due_to_timeout · cancelled_due_to_limits · cancelled_by_user

Response:

{
  "jobs": [
    {
      "job_id": "c7f8s2d9-...",
      "url": "https://example.com/",
      "status": "completed",
      "created_at": "2026-03-25T00:00:00+00:00",
      "updated_at": "2026-03-25T00:01:00+00:00"
    }
  ],
  "count": 1
}

Job Database

Jobs are persisted in a local SQLite database across server restarts.

Default location: ~/.local/share/mcp-cloudflare-crawl/jobs.db

Override with environment variable:

MCP_DB_PATH=/path/to/custom/jobs.db

Development

# Install dependencies
uv sync

# Run tests
uv run pytest

# Run tests with verbose output
uv run pytest -v

Notes

  • The Cloudflare Crawl API is asynchronouscrawl_start returns immediately, results are retrieved via crawl_status.

  • The crawler respects robots.txt by default. Disallowed URLs appear with "status": "disallowed".

  • The json format uses Workers AI for structured extraction and incurs additional charges.

  • Setting render: false skips the headless browser and fetches static HTML — faster and currently unbilled during beta.

  • Results are retained for 14 days after a job completes. Maximum job runtime is 7 days.

  • The crawler identifies itself as CloudflareBrowserRenderingCrawler/1.0 and cannot bypass Cloudflare protection or CAPTCHAs.

  • HTTP 429 (rate limit) responses are automatically retried with exponential backoff (up to 3 retries: 1s → 2s → 4s). The Retry-After response header is respected when present.

License

MIT

Available Tools

5 tools
crawl_and_waitA

Start a crawl and wait for it to complete, returning the final results.

This is a convenience tool that combines crawl_start and crawl_status polling. Suitable for small crawls (few pages). For large crawls, use crawl_start and crawl_status separately to avoid timeout issues.

Args: url: The starting URL to crawl (required). limit: Maximum pages to crawl (default: 10, max: 100000). depth: Maximum link depth (default: 100000). source: URL discovery source — "all", "sitemaps", or "links" (default: "all"). formats: Output formats — any of ["html", "markdown", "json"] (default: ["html"]). Note: "json" uses Workers AI and incurs additional charges. render: Whether to execute JavaScript via headless browser (default: true). max_age: Cache duration in seconds (default: 86400, max: 604800). modified_since: Unix timestamp — only crawl pages modified since this time. crawl_purposes: Declare content use — any of ["search", "ai-input", "ai-train"]. include_patterns: URL patterns to include (* = any chars except /, ** = any chars). exclude_patterns: URL patterns to exclude (takes priority over include_patterns). include_external_links: Whether to follow links to external domains. include_subdomains: Whether to follow links to subdomains. authenticate: HTTP authentication credentials for protected sites. Example: {"username": "user", "password": "pass"}. extra_http_headers: Custom HTTP headers to send with each crawl request. Example: {"X-API-Key": "abc123"}. json_options: AI-based structured data extraction config (requires "json" in formats). Keys: "prompt" (str) — extraction instruction, "response_format" (dict) — JSON schema for output, "custom_ai" (dict) — custom AI model config. cookies: Browser cookies to set during the crawl. Example: [{"name": "session", "value": "abc", "domain": "example.com"}]. goto_options: Page navigation behaviour. Keys: "waitUntil" (str) — e.g. "networkidle2", "load", "domcontentloaded"; "timeout" (int) — navigation timeout in milliseconds. wait_for_selector: Wait for a DOM element before scraping each page. Keys: "selector" (str), "timeout" (int, ms), "visible" (bool). reject_resource_types: Resource types to block to speed up crawls and reduce cost. Values: "image", "media", "font", "stylesheet", "script", etc. poll_interval: Seconds between status polls (default: 5.0). timeout: Maximum seconds to wait for completion (default: 300.0).

Returns: Final crawl result (same shape as crawl_status) once the job completes, or raises RuntimeError if the timeout is exceeded.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
depthNo
limitNo
renderNo
sourceNo
cookiesNo
formatsNo
max_ageNo
timeoutNo
authenticateNo
goto_optionsNo
json_optionsNo
poll_intervalNo
crawl_purposesNo
modified_sinceNo
exclude_patternsNo
include_patternsNo
wait_for_selectorNo
extra_http_headersNo
include_subdomainsNo
reject_resource_typesNo
include_external_linksNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does so fairly well: it states the blocking/polling behavior (poll_interval, timeout defaults), discloses that exceeding the timeout raises RuntimeError, and warns that 'json' formats use Workers AI and incur additional charges. It does not mention auth requirements or the exact failure modes beyond timeout, keeping it just short of a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The prose is front-loaded and tight, and the long Args block is justified by 22 undocumented parameters, each receiving a single compact line. Slightly verbose in places (multiple multi-line nested-key explanations), but there is essentially no wasted sentence.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a high-complexity, high-parameter tool, the description covers purpose, sibling routing, every parameter, cost implications, and the return/error contract ('same shape as crawl_status', RuntimeError on timeout). An output schema exists, so return-value detail is appropriately left out; nothing an agent needs to invoke it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% across 22 params, so the description must compensate entirely — and it does, documenting defaults, ranges (limit max 100000, max_age max 604800), enum values (source, formats, crawl_purposes, reject_resource_types), priority rules (exclude_patterns 'takes priority'), pattern syntax (* vs **), nested keys for json_options/goto_options/wait_for_selector, and concrete examples for authenticate/extra_http_headers/cookies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific verb+resource ('Start a crawl and wait for it to complete, returning the final results') and explicitly identifies itself as a composition of crawl_start and crawl_status, which cleanly distinguishes it from those siblings. An agent can tell what it does and how it relates to the other crawl tools without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit when/when-not: 'Suitable for small crawls (few pages). For large crawls, use crawl_start and crawl_status separately to avoid timeout issues.' That names both the alternative tools and the condition that selects them, leaving nothing to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_cancelB

Cancel a running crawl job.

Args: job_id: The crawl job ID returned by crawl_start.

Returns: {"success": true, "job_id": ""}

ParametersJSON Schema
NameRequiredDescriptionDefault
job_idYes

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.3/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full behavioral burden. It discloses the return shape but says nothing about idempotency, whether a finished job can be cancelled, whether cancellation is immediate, or permission requirements — significant gaps for a mutation tool.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the action, then scoped Args and Returns sections. Efficient, though the Returns line partially duplicates the existing output schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The output schema means return values need not be explained, and the single parameter is covered. But for a mutation with no annotations, the description omits operational semantics (idempotency, post-cancel state) an agent would want.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate, and it does: it identifies job_id as 'the crawl job ID returned by crawl_start,' giving the agent both meaning and provenance for the sole parameter.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Cancel a running crawl job') that cleanly distinguishes it from siblings like crawl_start, crawl_status, and crawl_list. No explicit sibling reference is needed given the distinct action.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'running crawl job' implies a precondition (only an in-flight job can be cancelled), which is mild usage guidance. However, there is no explicit when-to-use/when-not guidance, no mention of what happens if the job already finished, and no routing to alternatives like crawl_and_wait.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_listA

List all crawl jobs stored in the local database.

Jobs are recorded automatically when crawl_start or crawl_and_wait is called. Status is updated whenever crawl_status is polled.

Args: status_filter: Filter by job status — one of: "submitted", "running", "completed", "errored", "cancelled_due_to_timeout", "cancelled_due_to_limits", "cancelled_by_user". limit: Maximum number of jobs to return (default: 50). offset: Number of jobs to skip for pagination (default: 0).

Returns: { "jobs": [ { "job_id": "...", "url": "https://...", "status": "completed", "created_at": "2026-03-25T00:00:00+00:00", "updated_at": "2026-03-25T00:01:00+00:00" }, ... ], "count": }

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNo
offsetNo
status_filterNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does disclose meaningful behavior: the data source (local database), the lifecycle that populates it, and pagination defaults. It stops short of stating read-only semantics explicitly or any concurrency/rate behavior, but the listing nature makes the safety profile inferable.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the one-line purpose, then lifecycle context, then parameters. The Args section is dense and every line adds information not present in the schema's bare titles.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Parameters and lifecycle are completely covered, and for a simple list tool that is nearly everything an agent needs. The Returns block duplicates an existing output schema, which is redundant rather than harmful, and no filtering-by-URL or ordering behavior is mentioned.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must compensate, and it does fully: it enumerates every valid status_filter value (which the schema does not), and documents limit/offset defaults and pagination purpose. Nothing about the parameters is left undocumented.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ("List all crawl jobs stored in the local database"), which clearly separates it from action siblings like crawl_start or crawl_cancel. It doesn't explicitly name a sibling it overlaps with (e.g., crawl_status), but the purpose is unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explains the data model context that tells an agent when this is useful: jobs exist only after crawl_start/crawl_and_wait, and status is refreshed by polling crawl_status. It gives clear context but no explicit when-not or alternative-selection guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_startA

Start an asynchronous crawl job using Cloudflare's Browser Rendering Crawl API.

Submits a crawl job and returns a job_id immediately. Use crawl_status to poll for results, or use crawl_and_wait to block until completion.

Args: url: The starting URL to crawl (required). limit: Maximum number of pages to crawl (default: 10, max: 100000). depth: Maximum link depth to follow (default: 100000). source: URL discovery source — "all", "sitemaps", or "links" (default: "all"). formats: Output formats — any of ["html", "markdown", "json"] (default: ["html"]). Note: "json" uses Workers AI and incurs additional charges. render: Whether to execute JavaScript via headless browser (default: true). Set false for faster, unbilled static HTML fetching. max_age: Cache duration in seconds (default: 86400, max: 604800). modified_since: Unix timestamp — only crawl pages modified since this time. crawl_purposes: Declare content use — any of ["search", "ai-input", "ai-train"]. include_patterns: URL patterns to include (* = any chars except /, ** = any chars). exclude_patterns: URL patterns to exclude (takes priority over include_patterns). include_external_links: Whether to follow links to external domains. include_subdomains: Whether to follow links to subdomains. authenticate: HTTP authentication credentials for protected sites. Example: {"username": "user", "password": "pass"}. extra_http_headers: Custom HTTP headers to send with each crawl request. Example: {"X-API-Key": "abc123"}. json_options: AI-based structured data extraction config (requires "json" in formats). Keys: "prompt" (str) — extraction instruction, "response_format" (dict) — JSON schema for output, "custom_ai" (dict) — custom AI model config. cookies: Browser cookies to set during the crawl. Example: [{"name": "session", "value": "abc", "domain": "example.com"}]. goto_options: Page navigation behaviour. Keys: "waitUntil" (str) — e.g. "networkidle2", "load", "domcontentloaded"; "timeout" (int) — navigation timeout in milliseconds. wait_for_selector: Wait for a DOM element before scraping each page. Keys: "selector" (str), "timeout" (int, ms), "visible" (bool). reject_resource_types: Resource types to block to speed up crawls and reduce cost. Values: "image", "media", "font", "stylesheet", "script", etc.

Returns: {"job_id": ""} — use this ID with crawl_status or crawl_cancel.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
depthNo
limitNo
renderNo
sourceNo
cookiesNo
formatsNo
max_ageNo
authenticateNo
goto_optionsNo
json_optionsNo
crawl_purposesNo
modified_sinceNo
exclude_patternsNo
include_patternsNo
wait_for_selectorNo
extra_http_headersNo
include_subdomainsNo
reject_resource_typesNo
include_external_linksNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does well: it discloses that the job is asynchronous and returns immediately, that 'json' format incurs Workers AI charges, that render=false is faster and unbilled, that exclude_patterns take priority over include_patterns, and that reject_resource_types speeds up crawls. It omits rate limits, credential/auth requirements for the API itself, and error/failure behavior, so it falls short of exhaustive.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads purpose, async contract, and sibling routing in the first two sentences, then uses a structured Args/Returns layout. Given 20 parameters and 0% schema coverage, the length is justified and every line carries distinct information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need little explanation, and the description still briefly states the shape ({job_id}) and its use. Combined with the thorough parameter documentation and sibling routing, an agent has everything needed to invoke this correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, yet the description documents all 20 parameters with defaults, valid ranges (limit max 100000, max_age max 604800), allowed enum-like values ('all'/'sitemaps'/'links', formats), priority rules, and concrete examples for nested objects like authenticate, cookies, and goto_options. It fully compensates for the empty schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Start an asynchronous crawl job using Cloudflare's Browser Rendering Crawl API') and immediately clarifies the async contract by noting it returns a job_id. It distinguishes itself from siblings by naming crawl_status and crawl_and_wait as the follow-up tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly routes the agent: 'Use crawl_status to poll for results, or use crawl_and_wait to block until completion,' and mentions crawl_cancel in the Returns section. The alternative-selection conditions are stated rather than left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_statusA

Check the status and retrieve results of a crawl job.

For large result sets (>10 MB), the response includes a "cursor" value. Pass it back in the next call to paginate through results.

Args: job_id: The crawl job ID returned by crawl_start. cursor: Pagination token from a previous response (for large result sets). limit: Number of records to return per page. status_filter: Filter records by status — one of: "queued", "completed", "disallowed", "skipped", "errored", "cancelled".

Returns: { "id": "", "status": "running|completed|errored|cancelled_due_to_timeout|cancelled_due_to_limits|cancelled_by_user", "total": , "finished": , "browser_seconds_used": , "cursor": <int|null>, "records": [ { "url": "...", "status": "completed|errored|queued|disallowed|skipped|cancelled", "html": "...", # if html format requested "markdown": "...", # if markdown format requested "metadata": {"status": 200, "title": "...", "url": "..."} }, ... ] }

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNo
cursorNo
job_idYes
status_filterNo

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries the full burden, and it does disclose a genuine behavioral trait beyond structured fields: results over 10 MB return a cursor that must be passed back to paginate. It also enumerates the job-level status values (including timeout/limit cancellations), though it omits auth requirements, error behavior for unknown job_ids, and rate limits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with a one-line purpose, then pagination behavior, then Args and Returns sections — well structured and easy to scan. The large Returns block is somewhat verbose given an output schema already exists, but it is clearly delineated and each line carries information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so the description need not explain return values, yet it still documents pagination and all parameters completely for a 4-param tool. The only meaningful omissions are failure modes (invalid/expired job_id, what an errored job returns) and any permission requirements.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate entirely, and it does: job_id's origin, cursor's role as a pagination token from a prior response, limit as records-per-page, and status_filter's full closed set of six values. This is exactly the meaning the bare schema lacks.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Check the status and retrieve results of a crawl job') that an agent can act on immediately. It references crawl_start as the source of job_id, which gives partial routing help, but it does not distinguish itself from crawl_and_wait or crawl_list, which also relate to crawl jobs.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied rather than stated: job_id comes 'from crawl_start', so the tool is meant to be called after starting a job, and the cursor paragraph explains how to continue paging. There is no explicit when-to-use-vs-alternative guidance (e.g. versus crawl_and_wait) and no exclusions or prerequisites such as job lifetime or auth.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 5 tool updatesv0.1.0
    • First observedcrawl_and_wait
    • First observedcrawl_cancel
    • First observedcrawl_list
    • First observedcrawl_start
    • First observedcrawl_status

TDQS

A4/5.0

Scored across 5 tools

Disambiguation4/5

crawl_start, crawl_status, crawl_cancel, and crawl_list each have distinct purposes. crawl_and_wait overlaps with crawl_start+crawl_status polling, but its description clearly frames it as a convenience wrapper for small crawls, so confusion risk is limited.

Naming Consistency5/5

All tools follow the same crawl_ + verb/short-noun pattern (crawl_start, crawl_status, crawl_cancel, crawl_list, crawl_and_wait) in consistent snake_case.

Tool Count5/5

Five tools is well-scoped for an asynchronous crawl API, covering submission, polling, cancellation, listing, and a blocking convenience path without redundancy bloat.

Completeness4/5

Full job lifecycle is covered (start, status, cancel, list, and synchronous wait). Only minor gaps exist, such as deleting/clearing stored jobs or re-fetching a single job by ID.

Maintenance

ActivityInactive
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • F
    license
    Not graded
    quality
    C
    maintenance
    Enables AI assistants to control web browsers through Playwright automation tools deployed on Cloudflare Workers. Supports web navigation, clicking, typing, screenshot capture, and other browser automation tasks through natural language commands.
    -
  • F
    license
    Not graded
    quality
    C
    maintenance
    Enables AI assistants to control a web browser through tools for navigation, clicking, typing, and capturing screenshots using Cloudflare Workers. It allows models to perform complex web automation tasks and interact with live websites through a set of 14 specialized tools.
    4,678 npm
    -
  • A
    license
    Not graded
    quality
    D
    maintenance
    Enables Claude to scrape and crawl websites via a self-hosted Firecrawl instance through the Model Context Protocol. Provides tools for single/multi-URL scraping, site mapping, and full-site crawling operations.
    22,445 npm
    1
    MIT