mcp-cloudflare-crawl
Provides tools to crawl websites using Cloudflare's Browser Rendering Crawl API, with support for authentication, custom headers, AI extraction, and job management.
Leverages Cloudflare Workers AI for structured data extraction during crawling (requires 'json' format).
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@mcp-cloudflare-crawlcrawl https://example.com and return markdown content"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
mcp-cloudflare-crawl
An MCP server that exposes Cloudflare's Browser Rendering Crawl API as tools for LLM clients.
Requirements
A Cloudflare account with Browser Rendering enabled
A Cloudflare API token with Browser Rendering - Edit permission
Related MCP server: Hyperbrowser MCP Server
Setup
cp .env.example .env
# Edit .env and fill in your credentials.env:
CLOUDFLARE_API_TOKEN=your_api_token_here
CLOUDFLARE_ACCOUNT_ID=your_account_id_hereRunning
stdio (default — for Claude Desktop and most MCP clients)
uv run mcp-cloudflare-crawlStreamable HTTP
uv run mcp-cloudflare-crawl --transport streamable-http
# Listens on http://127.0.0.1:8000/mcp by default
uv run mcp-cloudflare-crawl --transport streamable-http --host 0.0.0.0 --port 9000Claude Code Integration
claude mcp add \
--env CLOUDFLARE_API_TOKEN=your_api_token_here \
--env CLOUDFLARE_ACCOUNT_ID=your_account_id_here \
cloudflare-crawl \
-- uv run --directory /absolute/path/to/mcp-cloudflare-crawl mcp-cloudflare-crawlAdd --scope user to make it available across all projects:
claude mcp add --scope user \
--env CLOUDFLARE_API_TOKEN=your_api_token_here \
--env CLOUDFLARE_ACCOUNT_ID=your_account_id_here \
cloudflare-crawl \
-- uv run --directory /absolute/path/to/mcp-cloudflare-crawl mcp-cloudflare-crawlClaude Desktop Integration
Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):
{
"mcpServers": {
"cloudflare-crawl": {
"command": "uv",
"args": [
"run",
"--directory",
"/absolute/path/to/mcp-cloudflare-crawl",
"mcp-cloudflare-crawl"
],
"env": {
"CLOUDFLARE_API_TOKEN": "your_api_token_here",
"CLOUDFLARE_ACCOUNT_ID": "your_account_id_here"
}
}
}
}Testing with curl
The server uses SSE (Server-Sent Events) format. Responses look like:
event: message
data: {"jsonrpc":"2.0","id":1,"result":{...}}To parse with jq, extract the data: line first:
curl -s -X POST http://127.0.0.1:8000/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-H "mcp-session-id: $SESSION_ID" \
-d '{...}' \
| grep '^data:' | sed 's/^data: //' | jq .Step 1 — Initialize session and capture session ID
SESSION_ID=$(curl -s -D - -X POST http://127.0.0.1:8000/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-d '{
"jsonrpc": "2.0", "id": 1, "method": "initialize",
"params": {
"protocolVersion": "2024-11-05",
"capabilities": {},
"clientInfo": {"name": "curl-test", "version": "1.0"}
}
}' | grep -i '^mcp-session-id:' | awk '{print $2}' | tr -d '\r')
echo "Session ID: $SESSION_ID"Step 2 — List available tools
curl -s -X POST http://127.0.0.1:8000/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-H "mcp-session-id: $SESSION_ID" \
-d '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}' \
| grep '^data:' | sed 's/^data: //' | jq .Step 3 — Start a crawl (all optional parameters)
curl -s -X POST http://127.0.0.1:8000/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-H "mcp-session-id: $SESSION_ID" \
-d '{
"jsonrpc": "2.0", "id": 3, "method": "tools/call",
"params": {
"name": "crawl_start",
"arguments": {
"url": "https://www.exampledocs.com/docs/",
"crawl_purposes": ["search"],
"limit": 50,
"depth": 2,
"formats": ["markdown"],
"render": true,
"max_age": 7200,
"source": "all",
"include_external_links": true,
"include_subdomains": true,
"include_patterns": ["**/api/v1/*"],
"exclude_patterns": ["*/learning-paths/*"],
"reject_resource_types": ["image", "media", "font"],
"goto_options": {"waitUntil": "networkidle2", "timeout": 30000},
"wait_for_selector": {"selector": "#content", "timeout": 5000}
}
}
}' | grep '^data:' | sed 's/^data: //' | jq .Step 4 — Poll status
curl -s -X POST http://127.0.0.1:8000/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-H "mcp-session-id: $SESSION_ID" \
-d '{
"jsonrpc": "2.0", "id": 4, "method": "tools/call",
"params": {
"name": "crawl_status",
"arguments": {"job_id": "YOUR_JOB_ID"}
}
}' | grep '^data:' | sed 's/^data: //' | jq .Step 5 — Crawl with AI structured extraction
Requires "json" in formats. Uses Cloudflare Workers AI and incurs additional charges.
curl -s -X POST http://127.0.0.1:8000/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-H "mcp-session-id: $SESSION_ID" \
-d '{
"jsonrpc": "2.0", "id": 5, "method": "tools/call",
"params": {
"name": "crawl_and_wait",
"arguments": {
"url": "https://example.com/",
"formats": ["json"],
"limit": 5,
"json_options": {
"prompt": "Extract product names and prices",
"response_format": {"type": "object"}
},
"timeout": 120.0
}
}
}' | grep '^data:' | sed 's/^data: //' | jq .Step 6 — Crawl a password-protected site
curl -s -X POST http://127.0.0.1:8000/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-H "mcp-session-id: $SESSION_ID" \
-d '{
"jsonrpc": "2.0", "id": 6, "method": "tools/call",
"params": {
"name": "crawl_start",
"arguments": {
"url": "https://internal.example.com/docs/",
"authenticate": {"username": "user", "password": "pass"},
"extra_http_headers": {"X-API-Key": "abc123"},
"cookies": [{"name": "session", "value": "xyz", "domain": "internal.example.com"}],
"formats": ["markdown"]
}
}
}' | grep '^data:' | sed 's/^data: //' | jq .Step 7 — List all stored jobs
curl -s -X POST http://127.0.0.1:8000/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-H "mcp-session-id: $SESSION_ID" \
-d '{
"jsonrpc": "2.0", "id": 7, "method": "tools/call",
"params": {
"name": "crawl_list",
"arguments": {}
}
}' | grep '^data:' | sed 's/^data: //' | jq .Tools
crawl_start
Submit a crawl job. Returns a job_id immediately — crawling happens asynchronously.
The job is automatically saved to the local SQLite database.
Parameter | Type | Description |
| string | Required. Starting URL to crawl |
| int | Max pages to crawl (default: 10, max: 100,000) |
| int | Max link depth (default: 100,000) |
| string | URL discovery: |
| list[string] | Output formats: |
| bool | Execute JavaScript via headless browser (default: true) |
| int | Cache duration in seconds (default: 86400, max: 604800) |
| int | Unix timestamp — only crawl pages modified since then |
| list[string] | Declare use: |
| list[string] | URL patterns to include ( |
| list[string] | URL patterns to exclude (higher priority than include) |
| bool | Follow links to external domains |
| bool | Follow links to subdomains |
| dict | HTTP auth credentials: |
| dict | Custom request headers: |
| dict | AI extraction config (requires |
| list[dict] | Browser cookies: |
| dict | Navigation behaviour: |
| dict | Wait for DOM element: |
| list[string] | Block resource types: |
Response:
{ "job_id": "c7f8s2d9-a8e7-4b6e-8e4d-3d4a1b2c3f4e" }crawl_status
Poll the status and results of a crawl job. Also updates the job's status in the local database.
Parameter | Type | Description |
| string | Required. Job ID from |
| int | Pagination token for large result sets (>10 MB) |
| int | Records per page |
| string | Filter by record status: |
Response:
{
"id": "c7f8s2d9-...",
"status": "completed",
"total": 20,
"finished": 20,
"browser_seconds_used": 134.7,
"cursor": null,
"records": [
{
"url": "https://example.com/",
"status": "completed",
"markdown": "# Example Domain\n...",
"metadata": { "status": 200, "title": "Example Domain", "url": "https://example.com/" }
}
]
}Job statuses: running · completed · errored · cancelled_due_to_timeout · cancelled_due_to_limits · cancelled_by_user
Record statuses: queued · completed · errored · disallowed · skipped · cancelled
crawl_cancel
Cancel a running crawl job.
Parameter | Type | Description |
| string | Required. Job ID from |
crawl_and_wait
Start a crawl and block until it completes, returning the final results. Combines crawl_start and crawl_status polling in one call. The job is saved and status is updated in the local database throughout.
Accepts all parameters from crawl_start, plus:
Parameter | Type | Description |
| float | Seconds between status polls (default: 5.0) |
| float | Max seconds to wait (default: 300.0) |
Use this for small crawls (a few pages). For large crawls, use crawl_start + crawl_status separately to avoid timeouts.
crawl_list
List all crawl jobs stored in the local SQLite database. Jobs are recorded automatically on crawl_start and crawl_and_wait, and their status is updated on every crawl_status or crawl_cancel call.
Parameter | Type | Description |
| string | Filter by job status (see below) |
| int | Max jobs to return (default: 50) |
| int | Jobs to skip for pagination (default: 0) |
Job statuses: submitted · running · completed · errored · cancelled_due_to_timeout · cancelled_due_to_limits · cancelled_by_user
Response:
{
"jobs": [
{
"job_id": "c7f8s2d9-...",
"url": "https://example.com/",
"status": "completed",
"created_at": "2026-03-25T00:00:00+00:00",
"updated_at": "2026-03-25T00:01:00+00:00"
}
],
"count": 1
}Job Database
Jobs are persisted in a local SQLite database across server restarts.
Default location: ~/.local/share/mcp-cloudflare-crawl/jobs.db
Override with environment variable:
MCP_DB_PATH=/path/to/custom/jobs.dbDevelopment
# Install dependencies
uv sync
# Run tests
uv run pytest
# Run tests with verbose output
uv run pytest -vNotes
The Cloudflare Crawl API is asynchronous —
crawl_startreturns immediately, results are retrieved viacrawl_status.The crawler respects
robots.txtby default. Disallowed URLs appear with"status": "disallowed".The
jsonformat uses Workers AI for structured extraction and incurs additional charges.Setting
render: falseskips the headless browser and fetches static HTML — faster and currently unbilled during beta.Results are retained for 14 days after a job completes. Maximum job runtime is 7 days.
The crawler identifies itself as
CloudflareBrowserRenderingCrawler/1.0and cannot bypass Cloudflare protection or CAPTCHAs.HTTP 429 (rate limit) responses are automatically retried with exponential backoff (up to 3 retries: 1s → 2s → 4s). The
Retry-Afterresponse header is respected when present.
License
Available Tools
5 toolscrawl_and_waitA
Start a crawl and wait for it to complete, returning the final results.
This is a convenience tool that combines crawl_start and crawl_status polling. Suitable for small crawls (few pages). For large crawls, use crawl_start and crawl_status separately to avoid timeout issues.
Args: url: The starting URL to crawl (required). limit: Maximum pages to crawl (default: 10, max: 100000). depth: Maximum link depth (default: 100000). source: URL discovery source — "all", "sitemaps", or "links" (default: "all"). formats: Output formats — any of ["html", "markdown", "json"] (default: ["html"]). Note: "json" uses Workers AI and incurs additional charges. render: Whether to execute JavaScript via headless browser (default: true). max_age: Cache duration in seconds (default: 86400, max: 604800). modified_since: Unix timestamp — only crawl pages modified since this time. crawl_purposes: Declare content use — any of ["search", "ai-input", "ai-train"]. include_patterns: URL patterns to include (* = any chars except /, ** = any chars). exclude_patterns: URL patterns to exclude (takes priority over include_patterns). include_external_links: Whether to follow links to external domains. include_subdomains: Whether to follow links to subdomains. authenticate: HTTP authentication credentials for protected sites. Example: {"username": "user", "password": "pass"}. extra_http_headers: Custom HTTP headers to send with each crawl request. Example: {"X-API-Key": "abc123"}. json_options: AI-based structured data extraction config (requires "json" in formats). Keys: "prompt" (str) — extraction instruction, "response_format" (dict) — JSON schema for output, "custom_ai" (dict) — custom AI model config. cookies: Browser cookies to set during the crawl. Example: [{"name": "session", "value": "abc", "domain": "example.com"}]. goto_options: Page navigation behaviour. Keys: "waitUntil" (str) — e.g. "networkidle2", "load", "domcontentloaded"; "timeout" (int) — navigation timeout in milliseconds. wait_for_selector: Wait for a DOM element before scraping each page. Keys: "selector" (str), "timeout" (int, ms), "visible" (bool). reject_resource_types: Resource types to block to speed up crawls and reduce cost. Values: "image", "media", "font", "stylesheet", "script", etc. poll_interval: Seconds between status polls (default: 5.0). timeout: Maximum seconds to wait for completion (default: 300.0).
Returns: Final crawl result (same shape as crawl_status) once the job completes, or raises RuntimeError if the timeout is exceeded.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| depth | No | ||
| limit | No | ||
| render | No | ||
| source | No | ||
| cookies | No | ||
| formats | No | ||
| max_age | No | ||
| timeout | No | ||
| authenticate | No | ||
| goto_options | No | ||
| json_options | No | ||
| poll_interval | No | ||
| crawl_purposes | No | ||
| modified_since | No | ||
| exclude_patterns | No | ||
| include_patterns | No | ||
| wait_for_selector | No | ||
| extra_http_headers | No | ||
| include_subdomains | No | ||
| reject_resource_types | No | ||
| include_external_links | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does so fairly well: it states the blocking/polling behavior (poll_interval, timeout defaults), discloses that exceeding the timeout raises RuntimeError, and warns that 'json' formats use Workers AI and incur additional charges. It does not mention auth requirements or the exact failure modes beyond timeout, keeping it just short of a 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The prose is front-loaded and tight, and the long Args block is justified by 22 undocumented parameters, each receiving a single compact line. Slightly verbose in places (multiple multi-line nested-key explanations), but there is essentially no wasted sentence.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a high-complexity, high-parameter tool, the description covers purpose, sibling routing, every parameter, cost implications, and the return/error contract ('same shape as crawl_status', RuntimeError on timeout). An output schema exists, so return-value detail is appropriately left out; nothing an agent needs to invoke it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0% across 22 params, so the description must compensate entirely — and it does, documenting defaults, ranges (limit max 100000, max_age max 604800), enum values (source, formats, crawl_purposes, reject_resource_types), priority rules (exclude_patterns 'takes priority'), pattern syntax (* vs **), nested keys for json_options/goto_options/wait_for_selector, and concrete examples for authenticate/extra_http_headers/cookies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with a specific verb+resource ('Start a crawl and wait for it to complete, returning the final results') and explicitly identifies itself as a composition of crawl_start and crawl_status, which cleanly distinguishes it from those siblings. An agent can tell what it does and how it relates to the other crawl tools without opening any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives an explicit when/when-not: 'Suitable for small crawls (few pages). For large crawls, use crawl_start and crawl_status separately to avoid timeout issues.' That names both the alternative tools and the condition that selects them, leaving nothing to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_cancelB
Cancel a running crawl job.
Args: job_id: The crawl job ID returned by crawl_start.
Returns: {"success": true, "job_id": ""}
| Name | Required | Description | Default |
|---|---|---|---|
| job_id | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden. It discloses the return shape but says nothing about idempotency, whether a finished job can be cancelled, whether cancellation is immediate, or permission requirements — significant gaps for a mutation tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the action, then scoped Args and Returns sections. Efficient, though the Returns line partially duplicates the existing output schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The output schema means return values need not be explained, and the single parameter is covered. But for a mutation with no annotations, the description omits operational semantics (idempotency, post-cancel state) an agent would want.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it does: it identifies job_id as 'the crawl job ID returned by crawl_start,' giving the agent both meaning and provenance for the sole parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Cancel a running crawl job') that cleanly distinguishes it from siblings like crawl_start, crawl_status, and crawl_list. No explicit sibling reference is needed given the distinct action.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The phrase 'running crawl job' implies a precondition (only an in-flight job can be cancelled), which is mild usage guidance. However, there is no explicit when-to-use/when-not guidance, no mention of what happens if the job already finished, and no routing to alternatives like crawl_and_wait.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_listA
List all crawl jobs stored in the local database.
Jobs are recorded automatically when crawl_start or crawl_and_wait is called. Status is updated whenever crawl_status is polled.
Args: status_filter: Filter by job status — one of: "submitted", "running", "completed", "errored", "cancelled_due_to_timeout", "cancelled_due_to_limits", "cancelled_by_user". limit: Maximum number of jobs to return (default: 50). offset: Number of jobs to skip for pagination (default: 0).
Returns: { "jobs": [ { "job_id": "...", "url": "https://...", "status": "completed", "created_at": "2026-03-25T00:00:00+00:00", "updated_at": "2026-03-25T00:01:00+00:00" }, ... ], "count": }
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| offset | No | ||
| status_filter | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does disclose meaningful behavior: the data source (local database), the lifecycle that populates it, and pagination defaults. It stops short of stating read-only semantics explicitly or any concurrency/rate behavior, but the listing nature makes the safety profile inferable.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the one-line purpose, then lifecycle context, then parameters. The Args section is dense and every line adds information not present in the schema's bare titles.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Parameters and lifecycle are completely covered, and for a simple list tool that is nearly everything an agent needs. The Returns block duplicates an existing output schema, which is redundant rather than harmful, and no filtering-by-URL or ordering behavior is mentioned.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate, and it does fully: it enumerates every valid status_filter value (which the schema does not), and documents limit/offset defaults and pagination purpose. Nothing about the parameters is left undocumented.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ("List all crawl jobs stored in the local database"), which clearly separates it from action siblings like crawl_start or crawl_cancel. It doesn't explicitly name a sibling it overlaps with (e.g., crawl_status), but the purpose is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explains the data model context that tells an agent when this is useful: jobs exist only after crawl_start/crawl_and_wait, and status is refreshed by polling crawl_status. It gives clear context but no explicit when-not or alternative-selection guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_startA
Start an asynchronous crawl job using Cloudflare's Browser Rendering Crawl API.
Submits a crawl job and returns a job_id immediately. Use crawl_status to poll for results, or use crawl_and_wait to block until completion.
Args: url: The starting URL to crawl (required). limit: Maximum number of pages to crawl (default: 10, max: 100000). depth: Maximum link depth to follow (default: 100000). source: URL discovery source — "all", "sitemaps", or "links" (default: "all"). formats: Output formats — any of ["html", "markdown", "json"] (default: ["html"]). Note: "json" uses Workers AI and incurs additional charges. render: Whether to execute JavaScript via headless browser (default: true). Set false for faster, unbilled static HTML fetching. max_age: Cache duration in seconds (default: 86400, max: 604800). modified_since: Unix timestamp — only crawl pages modified since this time. crawl_purposes: Declare content use — any of ["search", "ai-input", "ai-train"]. include_patterns: URL patterns to include (* = any chars except /, ** = any chars). exclude_patterns: URL patterns to exclude (takes priority over include_patterns). include_external_links: Whether to follow links to external domains. include_subdomains: Whether to follow links to subdomains. authenticate: HTTP authentication credentials for protected sites. Example: {"username": "user", "password": "pass"}. extra_http_headers: Custom HTTP headers to send with each crawl request. Example: {"X-API-Key": "abc123"}. json_options: AI-based structured data extraction config (requires "json" in formats). Keys: "prompt" (str) — extraction instruction, "response_format" (dict) — JSON schema for output, "custom_ai" (dict) — custom AI model config. cookies: Browser cookies to set during the crawl. Example: [{"name": "session", "value": "abc", "domain": "example.com"}]. goto_options: Page navigation behaviour. Keys: "waitUntil" (str) — e.g. "networkidle2", "load", "domcontentloaded"; "timeout" (int) — navigation timeout in milliseconds. wait_for_selector: Wait for a DOM element before scraping each page. Keys: "selector" (str), "timeout" (int, ms), "visible" (bool). reject_resource_types: Resource types to block to speed up crawls and reduce cost. Values: "image", "media", "font", "stylesheet", "script", etc.
Returns: {"job_id": ""} — use this ID with crawl_status or crawl_cancel.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| depth | No | ||
| limit | No | ||
| render | No | ||
| source | No | ||
| cookies | No | ||
| formats | No | ||
| max_age | No | ||
| authenticate | No | ||
| goto_options | No | ||
| json_options | No | ||
| crawl_purposes | No | ||
| modified_since | No | ||
| exclude_patterns | No | ||
| include_patterns | No | ||
| wait_for_selector | No | ||
| extra_http_headers | No | ||
| include_subdomains | No | ||
| reject_resource_types | No | ||
| include_external_links | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does well: it discloses that the job is asynchronous and returns immediately, that 'json' format incurs Workers AI charges, that render=false is faster and unbilled, that exclude_patterns take priority over include_patterns, and that reject_resource_types speeds up crawls. It omits rate limits, credential/auth requirements for the API itself, and error/failure behavior, so it falls short of exhaustive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loads purpose, async contract, and sibling routing in the first two sentences, then uses a structured Args/Returns layout. Given 20 parameters and 0% schema coverage, the length is justified and every line carries distinct information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need little explanation, and the description still briefly states the shape ({job_id}) and its use. Combined with the thorough parameter documentation and sibling routing, an agent has everything needed to invoke this correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, yet the description documents all 20 parameters with defaults, valid ranges (limit max 100000, max_age max 604800), allowed enum-like values ('all'/'sitemaps'/'links', formats), priority rules, and concrete examples for nested objects like authenticate, cookies, and goto_options. It fully compensates for the empty schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Start an asynchronous crawl job using Cloudflare's Browser Rendering Crawl API') and immediately clarifies the async contract by noting it returns a job_id. It distinguishes itself from siblings by naming crawl_status and crawl_and_wait as the follow-up tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly routes the agent: 'Use crawl_status to poll for results, or use crawl_and_wait to block until completion,' and mentions crawl_cancel in the Returns section. The alternative-selection conditions are stated rather than left to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_statusA
Check the status and retrieve results of a crawl job.
For large result sets (>10 MB), the response includes a "cursor" value. Pass it back in the next call to paginate through results.
Args: job_id: The crawl job ID returned by crawl_start. cursor: Pagination token from a previous response (for large result sets). limit: Number of records to return per page. status_filter: Filter records by status — one of: "queued", "completed", "disallowed", "skipped", "errored", "cancelled".
Returns: { "id": "", "status": "running|completed|errored|cancelled_due_to_timeout|cancelled_due_to_limits|cancelled_by_user", "total": , "finished": , "browser_seconds_used": , "cursor": <int|null>, "records": [ { "url": "...", "status": "completed|errored|queued|disallowed|skipped|cancelled", "html": "...", # if html format requested "markdown": "...", # if markdown format requested "metadata": {"status": 200, "title": "...", "url": "..."} }, ... ] }
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| cursor | No | ||
| job_id | Yes | ||
| status_filter | No |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden, and it does disclose a genuine behavioral trait beyond structured fields: results over 10 MB return a cursor that must be passed back to paginate. It also enumerates the job-level status values (including timeout/limit cancellations), though it omits auth requirements, error behavior for unknown job_ids, and rate limits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with a one-line purpose, then pagination behavior, then Args and Returns sections — well structured and easy to scan. The large Returns block is somewhat verbose given an output schema already exists, but it is clearly delineated and each line carries information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so the description need not explain return values, yet it still documents pagination and all parameters completely for a 4-param tool. The only meaningful omissions are failure modes (invalid/expired job_id, what an errored job returns) and any permission requirements.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate entirely, and it does: job_id's origin, cursor's role as a pagination token from a prior response, limit as records-per-page, and status_filter's full closed set of six values. This is exactly the meaning the bare schema lacks.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Check the status and retrieve results of a crawl job') that an agent can act on immediately. It references crawl_start as the source of job_id, which gives partial routing help, but it does not distinguish itself from crawl_and_wait or crawl_list, which also relate to crawl jobs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied rather than stated: job_id comes 'from crawl_start', so the tool is meant to be called after starting a job, and the cursor paragraph explains how to continue paging. There is no explicit when-to-use-vs-alternative guidance (e.g. versus crawl_and_wait) and no exclusions or prerequisites such as job lifetime or auth.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
5 tool updates
v0.1.0- First observed
crawl_and_wait - First observed
crawl_cancel - First observed
crawl_list - First observed
crawl_start - First observed
crawl_status
TDQS
Scored across 5 tools
crawl_start, crawl_status, crawl_cancel, and crawl_list each have distinct purposes. crawl_and_wait overlaps with crawl_start+crawl_status polling, but its description clearly frames it as a convenience wrapper for small crawls, so confusion risk is limited.
All tools follow the same crawl_ + verb/short-noun pattern (crawl_start, crawl_status, crawl_cancel, crawl_list, crawl_and_wait) in consistent snake_case.
Five tools is well-scoped for an asynchronous crawl API, covering submission, polling, cancellation, listing, and a blocking convenience path without redundancy bloat.
Full job lifecycle is covered (start, status, cancel, list, and synchronous wait). Only minor gaps exist, such as deleting/clearing stored jobs or re-fetching a single job by ID.
Maintenance
Related MCP Connectors
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
Web search, browser automation, scraping, crawling and CAPTCHA solving for AI agents.
AI-powered browser automation — navigate, click, fill forms, and extract data from any website.
Enable language models to perform advanced AI-powered web scraping with enterprise-grade reliabili…
Related MCP Servers
- FlicenseNot gradedqualityCmaintenanceEnables AI assistants to control web browsers through Playwright automation tools deployed on Cloudflare Workers. Supports web navigation, clicking, typing, screenshot capture, and other browser automation tasks through natural language commands.-
- AlicenseNot gradedqualityNot gradedmaintenanceProvides tools to scrape, extract structured data, and crawl webpages, with access to browser automation agents like OpenAI's CUA, Anthropic's Claude Computer Use, and Browser Use for complex web tasks.-
- FlicenseNot gradedqualityCmaintenanceEnables AI assistants to control a web browser through tools for navigation, clicking, typing, and capturing screenshots using Cloudflare Workers. It allows models to perform complex web automation tasks and interact with live websites through a set of 14 specialized tools.4,678 npm-
- AlicenseNot gradedqualityDmaintenanceEnables Claude to scrape and crawl websites via a self-hosted Firecrawl instance through the Model Context Protocol. Provides tools for single/multi-URL scraping, site mapping, and full-site crawling operations.22,445 npm1MIT