mcp-cloudflare-crawl

crawl_start

Starts an asynchronous Cloudflare Browser Rendering crawl job and returns a job_id immediately. Poll it with crawl_status or wait for completion with crawl_and_wait.

Instructions

Start an asynchronous crawl job using Cloudflare's Browser Rendering Crawl API.

Submits a crawl job and returns a job_id immediately. Use crawl_status to poll for results, or use crawl_and_wait to block until completion.

Args: url: The starting URL to crawl (required). limit: Maximum number of pages to crawl (default: 10, max: 100000). depth: Maximum link depth to follow (default: 100000). source: URL discovery source — "all", "sitemaps", or "links" (default: "all"). formats: Output formats — any of ["html", "markdown", "json"] (default: ["html"]). Note: "json" uses Workers AI and incurs additional charges. render: Whether to execute JavaScript via headless browser (default: true). Set false for faster, unbilled static HTML fetching. max_age: Cache duration in seconds (default: 86400, max: 604800). modified_since: Unix timestamp — only crawl pages modified since this time. crawl_purposes: Declare content use — any of ["search", "ai-input", "ai-train"]. include_patterns: URL patterns to include (* = any chars except /, ** = any chars). exclude_patterns: URL patterns to exclude (takes priority over include_patterns). include_external_links: Whether to follow links to external domains. include_subdomains: Whether to follow links to subdomains. authenticate: HTTP authentication credentials for protected sites. Example: {"username": "user", "password": "pass"}. extra_http_headers: Custom HTTP headers to send with each crawl request. Example: {"X-API-Key": "abc123"}. json_options: AI-based structured data extraction config (requires "json" in formats). Keys: "prompt" (str) — extraction instruction, "response_format" (dict) — JSON schema for output, "custom_ai" (dict) — custom AI model config. cookies: Browser cookies to set during the crawl. Example: [{"name": "session", "value": "abc", "domain": "example.com"}]. goto_options: Page navigation behaviour. Keys: "waitUntil" (str) — e.g. "networkidle2", "load", "domcontentloaded"; "timeout" (int) — navigation timeout in milliseconds. wait_for_selector: Wait for a DOM element before scraping each page. Keys: "selector" (str), "timeout" (int, ms), "visible" (bool). reject_resource_types: Resource types to block to speed up crawls and reduce cost. Values: "image", "media", "font", "stylesheet", "script", etc.

Returns: {"job_id": ""} — use this ID with crawl_status or crawl_cancel.

Input Schema

TableJSON Schema

Name	Required	Description	Default
`url`	Yes
`depth`	No
`limit`	No
`render`	No
`source`	No
`cookies`	No
`formats`	No
`max_age`	No
`authenticate`	No
`goto_options`	No
`json_options`	No
`crawl_purposes`	No
`modified_since`	No
`exclude_patterns`	No
`include_patterns`	No
`wait_for_selector`	No
`extra_http_headers`	No
`include_subdomains`	No
`reject_resource_types`	No
`include_external_links`	No

Output Schema

TableJSON Schema

Name	Required	Description	Default
No arguments

Schema Changelog

Changes observed during successful MCP inspections.

First observedv0.1.0Sep 13, 2026

TDQS

A4.8/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does well: it discloses that the job is asynchronous and returns immediately, that 'json' format incurs Workers AI charges, that render=false is faster and unbilled, that exclude_patterns take priority over include_patterns, and that reject_resource_types speeds up crawls. It omits rate limits, credential/auth requirements for the API itself, and error/failure behavior, so it falls short of exhaustive.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads purpose, async contract, and sibling routing in the first two sentences, then uses a structured Args/Returns layout. Given 20 parameters and 0% schema coverage, the length is justified and every line carries distinct information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need little explanation, and the description still briefly states the shape ({job_id}) and its use. Combined with the thorough parameter documentation and sibling routing, an agent has everything needed to invoke this correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, yet the description documents all 20 parameters with defaults, valid ranges (limit max 100000, max_age max 604800), allowed enum-like values ('all'/'sitemaps'/'links', formats), priority rules, and concrete examples for nested objects like authenticate, cookies, and goto_options. It fully compensates for the empty schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Start an asynchronous crawl job using Cloudflare's Browser Rendering Crawl API') and immediately clarifies the async contract by noting it returns a job_id. It distinguishes itself from siblings by naming crawl_status and crawl_and_wait as the follow-up tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly routes the agent: 'Use crawl_status to poll for results, or use crawl_and_wait to block until completion,' and mentions crawl_cancel in the Returns section. The alternative-selection conditions are stated rather than left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server