Use this MCP server to scrape, crawl, map, and extract structured data from websites, plus parse local PDFs.
Scrape a URL to markdown, HTML, links, or images, with options for JS rendering, renderer choice, CSS includes/excludes, main-content stripping, wait times, and max content length.
Crawl a site asynchronously with depth/page limits, JS rendering controls, and optional JSON Schema-based LLM extraction per page; returns a job ID to poll.
Check crawl status to retrieve discovered pages and their content, with optional max content length per page.
Map a site to discover URLs via sitemap and/or short crawl, returning only a URL list without page content.
Extract structured JSON from one or more URLs using a free-text prompt and/or JSON Schema, with optional per-field evidence (basis), BYOK LLM settings, and async job polling.
Check or cancel extract jobs to view status/results/per-URL data/token usage/credits or request cancellation.
Parse a local PDF from base64 to markdown, plain text, links, images, JSON, or summary; no OCR support.
curl -fsSL https://fastcrw.com/install | shRuns local and free, no account needed. To use the Cloud, paste your key into the same command and it installs the binary, connects the key, and registers the MCP server with the AI coding tools you already have:
curl -fsSL https://fastcrw.com/install | CRW_API_KEY=crw_live_... shcrw search "rust tutorials"Claude Code, Cursor, Codex, Gemini CLI, OpenCode and Windsurf are picked up
automatically when they are already set up; nothing else is touched, and your
key stays in ~/.config/crw/config.toml rather than being copied into each
tool. Add CRW_NO_AGENTS=1 to skip that step, or run crw setup on its own to
choose interactively.
1000 free credits, no credit card. Managed proxies, JS rendering and search, with nothing to run or keep up to date. Get my free key →
macOS and Linux, Intel and ARM. More install options →
Related MCP server: webpeel
Operation | Outcome |
Scrape | One URL to markdown, HTML, links, screenshots, or schema JSON |
Crawl | Follow a bounded site crawl and collect its pages |
Map | Discover URLs without scraping every page |
Search | Search the web and optionally scrape selected results |
Extract | Produce structured fields from one or many URLs |
On Firecrawl's own public 1,000-URL dataset, fastCRW recovered more truth than Crawl4AI and Firecrawl, matched the fastest median latency, and idled at ~14 MB RAM.
On a different benchmark entirely, answer accuracy rather than scrape recall, fastCRW answers 90.0% of the 600 AA-Omniscience questions correctly. Every product listed on the Artificial Analysis Search Index sits below it.
crw https://example.com # scrape, works right after install
crw search "rust async runtime" # search, after `crw setup`Using Cloud? Get an API key, then export it once:
export CRW_API_KEY="crw_live_..."
pip install crwfrom crw import CrwClient
client = CrwClient()
page = client.scrape("https://example.com", formats=["markdown"])
print(page["markdown"])npm install crw-sdkimport { CrwClient } from "crw-sdk";
const client = new CrwClient();
const page = await client.scrape("https://example.com", {
formats: ["markdown"],
});
console.log(page.markdown);Local mode and more SDK examples → · REST API →
npx -y crw-mcp@latest installInstalls the CRW skill and MCP server in your detected AI tools. crw setup can
also do this step, so either path is enough.
Manual setup →
Managed API | Local / self-hosted | |
Best for | Zero infrastructure and managed scaling | Data control, private networks, or custom infrastructure |
Start | Create an API key, then | Install and run |
Operations | Managed proxies, billing, and hosted capabilities | You choose renderers, search, auth, proxies, and capacity |
Capabilities and response shapes can differ by deployment:
/v1/capabilities · response shapes
The workspace requires Rust 1.85 or newer:
git clone https://github.com/us/crw
cd crw
make check-fastcrw is better because of these people. Thank you, all of you:
Engine and MCP server: AGPL-3.0. Python and TypeScript SDKs: MIT. Embedding license: hello@fastcrw.com.
Please respect website policies. Crawl and map follow robots.txt by default.
Request cancellation of an extract job. Returns the canonical status; cancelling remains non-terminal until the claimed URL settles.
| Name | Required | Description | Default |
|---|---|---|---|
| id | Yes | Extract job id from crw_extract |
| Name | Required | Description |
|---|---|---|
| id | Yes | |
| error | No | |
| status | Yes | |
| results | Yes | |
| success | Yes | |
| expiresAt | Yes | |
| tokensUsed | Yes | |
| creditsUsed | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Despite annotations already indicating destructive and non-read-only behavior, the description adds valuable context: cancellation is non-terminal until the claimed URL settles, and it returns canonical status. This goes beyond the structured annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tightly worded sentences. The first states the action; the second adds a critical caveat. No redundant information; front-loaded with purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a cancellation tool, the description covers purpose and the important async behavior. An output schema exists, so return values need not be described. Slightly more guidance on subsequent steps (e.g., checking status) could help, but it is not essential.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with the single 'id' parameter documented as 'Extract job id from crw_extract.' The description text itself adds no parameter details, so it relies entirely on the schema, which is adequate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states the action: 'Request cancellation of an extract job.' The verb 'cancel' and resource 'extract job' are specific, and it is distinct from sibling tools like crw_extract (create) and crw_check_extract_status (status check).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies use when you want to cancel an extract job, but it does not explicitly contrast with alternatives or mention when not to use it. No sibling tool is referenced, so guidance is implied rather than explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Poll an async crawl job and retrieve its pages.
| Name | Required | Description | Default |
|---|---|---|---|
| id | Yes | Crawl job id from crw_crawl | |
| maxLength | No | Max chars per page content field; 0 = unbounded (default ~15000) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint and idempotentHint, so the description adds only moderate behavioral context by confirming it polls and retrieves pages. The description does not disclose any additional traits beyond the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, concise sentence that is front-loaded and contains no unnecessary words. It efficiently conveys the tool's purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple polling tool with two parameters and no output schema, the description adequately explains the action (poll and retrieve pages). It could hint at the return format, but 'retrieve its pages' is sufficient for an agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, with clear parameter descriptions in the schema. The tool description adds no additional meaning beyond what is already in the schema, meeting the baseline for high coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool polls an async crawl job and retrieves its pages, providing a specific verb and resource. It distinguishes from siblings like crw_crawl (start) and crw_scrape (synchronous scrape).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage context by mentioning 'poll an async crawl job', indicating it's for checking ongoing crawls. However, it does not explicitly state when not to use or name alternatives, which is a minor gap.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Poll an extract job; returns status and, when complete, a per-URL results array.
| Name | Required | Description | Default |
|---|---|---|---|
| id | Yes | Extract job id from crw_extract |
| Name | Required | Description |
|---|---|---|
| id | Yes | |
| error | No | |
| status | Yes | |
| results | Yes | |
| success | Yes | |
| expiresAt | Yes | |
| tokensUsed | Yes | |
| creditsUsed | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations already indicate a safe, idempotent read operation. The description adds the behavioral detail that the tool returns status immediately and, upon completion, includes a per-URL results array, which is useful for understanding polling behavior. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence that directly states the action and outcome with no extraneous text. It earns a top score.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that an output schema exists and the tool has a single parameter, the description adequately covers the tool's purpose, return behavior, and usage context. It is complete for the tool's complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already defines 'id' as the extract job id from crw_extract, and the description does not add any additional parameter semantics. With 100% schema coverage, the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses the specific verb 'Poll' and identifies the resource as 'an extract job,' clearly distinguishing it from the similar sibling 'crw_check_crawl_status' by specifying 'extract' and the per-URL results array. It states exactly what the tool does and its output.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It clearly implies usage when checking the status of a previously submitted extract job, providing clear context. However, it does not explicitly mention alternatives or when not to use this tool, such as for crawl jobs, so it lacks explicit exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Start an async site crawl; returns a job id to poll with crw_check_crawl_status.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Starting URL | |
| waitFor | No | Ms to wait after JS render per page | |
| maxDepth | No | Max crawl depth (default 2) | |
| maxPages | No | Max pages to crawl (default 10) | |
| renderJs | No | Force JS render (true), HTTP-only (false), omit = auto | |
| renderer | No | Pin renderer; browser tiers imply renderJs:true (default auto). 'camoufox' needs the opt-in tier configured. 'impersonated-http' is JS-less Chrome-TLS impersonation, never renderJs. | |
| jsonSchema | No | Optional. A JSON Schema (draft 2020-12) describing fields to extract from each page via an LLM, e.g. {"type":"object","properties":{"title":{"type":"string"}}}. Free-form object. Omit to crawl without structured extraction. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate openWorldHint=true (external side effects) and idempotentHint=false. The description adds the important async behavior and the job-id return contract, which are not in the annotations. It does not contradict any annotation and provides a clear behavioral expectation, though it omits details like potential load on target site (covered by openWorldHint).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence that states the core action and the immediate return value. There is zero fluff, and the most important behavioral fact (async + polling) is delivered first.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 7 parameters and no output schema, the description covers the essential contract: it starts an async crawl and returns a job id for polling. It does not explain the meaning of parameters (schema does) nor the external side effects (annotations do). The only minor gap is a lack of explicit warning about resource usage or cost, but openWorldHint and the async nature cover that adequately.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%: every parameter (url, waitFor, maxDepth, maxPages, renderJs, renderer, jsonSchema) has a descriptive definition. The tool description itself mentions none of these parameters, so it adds no additional semantics beyond the schema. Baseline 3 is appropriate because the schema already carries the full burden.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb ('start'), a clear resource ('site crawl'), and immediately explains the async nature and the follow-up mechanic (poll with crw_check_crawl_status). This differentiates it from siblings like crw_scrape or crw_check_crawl_status without ambiguity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It clearly establishes the typical workflow: start crawl, then poll status via the named sibling. It does not explicitly list alternatives or when not to use (e.g., for single-page extraction, use crw_scrape), but the async + polling context effectively guides usage and distinguishes it from crw_check_crawl_status.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Extract structured JSON from URLs via a prompt and/or JSON schema. Async job — poll crw_check_extract_status with the returned id. Needs an LLM.
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | URLs to extract from | |
| basis | No | Return per-field evidence: each top-level scalar property comes back with a source url, verbatim excerpt and honest status (supported/unverified/unsupported/notFound). Requires schema. | |
| prompt | No | Free-text extraction objective (required unless schema is given) | |
| schema | No | JSON Schema constraining the extracted output | |
| llmModel | No | ||
| llmApiKey | No | BYOK LLM API key | |
| llmProvider | No |
| Name | Required | Description |
|---|---|---|
| id | Yes | |
| urls | Yes | |
| status | Yes | |
| success | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate readOnlyHint=false and openWorldHint=true. The description adds behavioral nuance by disclosing the async job nature ('Async job — poll crw_check_extract_status with the returned id') and a key prerequisite ('Needs an LLM'). This provides context beyond the annotations, such as the non-blocking execution model and the requirement to track progress via a returned identifier.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is exactly two sentences: the first states the core purpose and method, the second covers the async workflow and prerequisite. Every clause adds value, and the most important information is front-loaded. There is no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 7 parameters, nested objects, and an output schema, the description efficiently covers the essential workflow (async, polling, LLM requirement). The existence of an output schema means return values are documented elsewhere. It does not discuss error handling or rate limits, but these are less critical given the asynchronous pattern and available schema documentation. Overall, it is complete enough for an agent to select and invoke the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema covers 71% of parameters with descriptions, so the baseline is 3. The description adds meaningful semantics by explaining that extraction works 'via a prompt and/or JSON schema', clarifying the relationship between prompt and schema parameters. It also highlights the LLM dependency, tying together llmModel/llmProvider/llmApiKey even though the schema doesn't explicitly state they are required. This goes beyond the schema's bare parameter list.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description begins with 'Extract structured JSON from URLs via a prompt and/or JSON schema', which clearly states the verb ('extract'), the resource ('URLs'), and the output format ('structured JSON'). It also distinguishes from sibling tools like crw_scrape (raw scraping) and crw_map by emphasizing structured extraction. The async note differentiates it from synchronous tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides context that it is an async job and must be polled via crw_check_extract_status, and that it needs an LLM. However, it does not explicitly state when to use this tool over alternatives such as crw_scrape or crw_crawl, nor any exclusions. The usage context is clear but lacks explicit when-to-use/when-not-to-use guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Discover URLs on a site via sitemap and/or a short crawl. Returns a URL list only, no page content.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to map | |
| limit | No | Max URLs to discover AND return; 0 = unbounded (default 100). Raise it (e.g. 50000) to pull deep/large sitemaps. | |
| maxDepth | No | Max discovery depth (default 2) | |
| useSitemap | No | Use sitemap.xml (default true) | |
| crawlFallback | No | Supplement sitemap with a short BFS crawl (default true; false = sitemap-only) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already provide readOnly, idempotent, and non-destructive hints. The description adds that output is URL list only, consistent with annotations, but no extra behavioral details beyond annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise, front-loaded sentences with no wasted words; every part adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity and rich schema+annotations, the description covers the main action and output. Minor gap: interaction of sitemap and crawl fallback is explained in schema, so description is complete enough.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has 100% parameter coverage with clear descriptions (e.g., limit 0 = unbounded, defaults). The description adds no further parameter meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool discovers URLs on a site via sitemap/crawl and explicitly says it returns only URLs, no content, distinguishing it from sibling crw_scrape and implying it's different from crw_crawl.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for URL discovery without content, but does not explicitly contrast with siblings like crw_crawl or provide when-to-use/not-use guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Parse a local PDF (base64 in contentBase64) to markdown. No OCR: scanned PDFs return empty markdown with a warning.
| Name | Required | Description | Default |
|---|---|---|---|
| formats | No | Output formats (default ["markdown"]); json/summary need a server LLM | |
| parsers | No | Parsers to apply (default ["pdf"]) | |
| filename | No | Original filename (optional) | |
| maxLength | No | Max chars per content field; 0 = unbounded (default ~15000) | |
| jsonSchema | No | Optional. A JSON Schema (draft 2020-12) describing fields to extract when formats includes "json", e.g. {"type":"object","properties":{"title":{"type":"string"}}}. Free-form object. | |
| contentBase64 | Yes | Base64-encoded PDF bytes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false, so the safety profile is clear. The description adds the behavioral trait that no OCR is performed and scanned PDFs return empty markdown with a warning, which is valuable context beyond annotations. It also clarifies that the input is base64-encoded local PDF bytes, but that's already in schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is one sentence, front-loaded with the main action ('Parse a local PDF...to markdown') and then a brief caveat. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 6 parameters and no output schema, the description covers the core functionality and a key limitation. However, it doesn't describe the response structure or that other output formats (json, summary) require a server LLM, though that's in the schema. The description is adequate for a simple parse tool but leaves some gap regarding output shape.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema provides descriptions for all 6 parameters (100% coverage), so the description doesn't need to add much. The description does reference contentBase64 and the markdown output, implicitly mapping to formats, but it doesn't explain the formats, jsonSchema, or maxLength parameters – though those are well-documented in the schema. Thus, the description adds minimal additional parameter semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool parses a local PDF (base64 in contentBase64) to markdown, which is a specific verb with resource and output format. It distinguishes itself from sibling tools by specifying 'local PDF' rather than URLs, aligning with crw_scrape/crw_crawl. The 'No OCR' caveat further defines its scope.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use it: when you have a local PDF as base64 and want markdown. It explicitly states a when-not scenario: scanned PDFs return empty markdown with a warning, which tells the agent to avoid using it for those. However, it doesn't name alternative tools for OCR or other formats, so it falls just short of explicit alternatives guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Scrape one URL to markdown, HTML, or links.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to scrape | |
| formats | No | Output formats (default ["markdown"]) | |
| waitFor | No | Ms to wait after JS render for late content | |
| renderJs | No | Force JS render (true), HTTP-only (false), omit = auto | |
| renderer | No | Pin renderer; browser tiers imply renderJs:true (default auto). 'camoufox' needs the opt-in tier configured. 'impersonated-http' is JS-less Chrome-TLS impersonation, never renderJs. | |
| maxLength | No | Max chars per content field; 0 = unbounded (default ~15000) | |
| excludeTags | No | CSS selectors to exclude | |
| includeTags | No | CSS selectors to include | |
| onlyMainContent | No | Strip nav/footer; main content only (default true) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, openWorldHint, idempotentHint, and destructiveHint=false, covering the safety profile. The description adds output format context but does not mention rendering behavior, potential delays, or response format. It does not contradict annotations and provides modest additional info about what the tool produces.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that wastes no words and leads with the action and resource. It is appropriately concise for a tool with a well-documented schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 9 parameters, only one required, and no output schema, the description is minimal. It lacks usage guidance and does not mention the 'images' format even though it is in the enum. However, the schema descriptions are thorough, so the agent can infer parameter behavior. The description is adequate but leaves gaps in when-to-use and output details.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so each parameter has a description. The tool description does not add parameter-specific meaning beyond that, and it does not clarify ambiguous terms. Baseline of 3 is appropriate given the schema fully documents the parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Scrape one URL to markdown, HTML, or links' clearly identifies the verb (scrape), the resource (one URL), and the output types. It distinguishes from siblings like crw_crawl (multiple URLs) and crw_extract (structured data) by emphasizing a single URL and specific formats.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies single-page scraping but does not explicitly state when to prefer this over siblings like crw_crawl or crw_extract. There are no exclusions, alternatives, or conditions. The phrase 'one URL' hints at the scope, but no explicit guidance is provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.35.1crw_crawl2 fields changedInput schema / properties / renderer / descriptionPrevious value: -"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto). 'camoufox' requires the server's opt-in camoufox tier to be configured."New value: +"Pin renderer; browser tiers imply renderJs:true (default auto). 'camoufox' needs the opt-in tier configured. 'impersonated-http' is JS-less Chrome-TLS impersonation, never renderJs."Input schema / properties / renderer / enumPrevious value: -[
- "auto",
- "lightpanda",
- "chrome",
- "playwright",
- "camoufox"
-]New value: +[
+ "auto",
+ "lightpanda",
+ "chrome",
+ "playwright",
+ "camoufox",
+ "impersonated-http"
+]crw_scrape2 fields changedInput schema / properties / renderer / descriptionPrevious value: -"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto). 'camoufox' requires the server's opt-in camoufox tier to be configured."New value: +"Pin renderer; browser tiers imply renderJs:true (default auto). 'camoufox' needs the opt-in tier configured. 'impersonated-http' is JS-less Chrome-TLS impersonation, never renderJs."Input schema / properties / renderer / enumPrevious value: -[
- "auto",
- "lightpanda",
- "chrome",
- "playwright",
- "camoufox"
-]New value: +[
+ "auto",
+ "lightpanda",
+ "chrome",
+ "playwright",
+ "camoufox",
+ "impersonated-http"
+]v0.30.0crw_cancel_extractcrw_check_extract_status1 field changedOutput schema / (root)Previous value: -nullNew value: +{
+ "additionalProperties": false,
+ "properties": {
+ "creditsUsed": {
+ "type": "integer"
+ },
+ "error": {
+ "type": "string"
+ },
+ "expiresAt": {
+ "format": "date-time",
+ "type": "string"
+ },
+ "id": {
+ "type": "string"
+ },
+ "results": {
+ "items": {
+ "additionalProperties": false,
+ "properties": {
+ "basis": {
+ "items": {
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "basisWarnings": {
+ "items": {
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "data": {
+ "additionalProperties": true,
+ "type": "object"
+ },
+ "error": {
+ "type": "string"
+ },
+ "llmInputHash": {
+ "type": "string"
+ },
+ "llmUsage": {
+ "type": "object"
+ },
+ "status": {
+ "enum": [
+ "processing",
+ "completed",
+ "failed",
+ "cancelled"
+ ],
+ "type": "string"
+ },
+ "url": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "url",
+ "status"
+ ],
+ "type": "object"
+ },
+ "type": "array"
+ },
+ "status": {
+ "enum": [
+ "processing",
+ "cancelling",
+ "completed",
+ "failed",
+ "cancelled"
+ ],
+ "type": "string"
+ },
+ "success": {
+ "type": "boolean"
+ },
+ "tokensUsed": {
+ "type": "integer"
+ }
+ },
+ "required": [
+ "success",
+ "id",
+ "status",
+ "results",
+ "expiresAt",
+ "tokensUsed"
+ ],
+ "type": "object"
+}crw_extract1 field changedOutput schema / (root)Previous value: -nullNew value: +{
+ "additionalProperties": false,
+ "properties": {
+ "id": {
+ "type": "string"
+ },
+ "status": {
+ "enum": [
+ "processing"
+ ],
+ "type": "string"
+ },
+ "success": {
+ "type": "boolean"
+ },
+ "urls": {
+ "minimum": 0,
+ "type": "integer"
+ }
+ },
+ "required": [
+ "success",
+ "id",
+ "status",
+ "urls"
+ ],
+ "type": "object"
+}crw_parse_file1 field changedInput schema / properties / formats / items / enumPrevious value: -[
- "markdown",
- "plainText",
- "links",
- "json",
- "summary"
-]New value: +[
+ "markdown",
+ "plainText",
+ "links",
+ "images",
+ "json",
+ "summary"
+]crw_scrape1 field changedInput schema / properties / formats / items / enumPrevious value: -[
- "markdown",
- "html",
- "links"
-]New value: +[
+ "markdown",
+ "html",
+ "links",
+ "images"
+]v0.24.1crw_extract1 field changedInput schema / properties / basisAdded value: +{
+ "description": "Return per-field evidence: each top-level scalar property comes back with a source url, verbatim excerpt and honest status (supported/unverified/unsupported/notFound). Requires schema.",
+ "type": "boolean"
+}v0.22.0crw_check_extract_statuscrw_extractv1.0.1crw_map1 field changedInput schema / properties / limit / descriptionPrevious value: -"Max URLs returned; 0 = unbounded (default 100)"New value: +"Max URLs to discover AND return; 0 = unbounded (default 100). Raise it (e.g. 50000) to pull deep/large sitemaps."v0.18.0crw_crawl4 fields changedInput schema / properties / jsonSchema / additionalPropertiesAdded value: +trueInput schema / properties / jsonSchema / descriptionPrevious value: -"JSON schema for LLM extraction per page"New value: +"Optional. A JSON Schema (draft 2020-12) describing fields to extract from each page via an LLM, e.g. {\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"}}}. Free-form object. Omit to crawl without structured extraction."Input schema / properties / renderer / descriptionPrevious value: -"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto)"New value: +"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto). 'camoufox' requires the server's opt-in camoufox tier to be configured."Input schema / properties / renderer / enumPrevious value: -[
- "auto",
- "lightpanda",
- "chrome",
- "playwright"
-]New value: +[
+ "auto",
+ "lightpanda",
+ "chrome",
+ "playwright",
+ "camoufox"
+]crw_parse_file2 fields changedInput schema / properties / jsonSchema / additionalPropertiesAdded value: +trueInput schema / properties / jsonSchema / descriptionPrevious value: -"JSON schema for LLM extraction (when formats has json)"New value: +"Optional. A JSON Schema (draft 2020-12) describing fields to extract when formats includes \"json\", e.g. {\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"}}}. Free-form object."crw_scrape2 fields changedInput schema / properties / renderer / descriptionPrevious value: -"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto)"New value: +"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto). 'camoufox' requires the server's opt-in camoufox tier to be configured."Input schema / properties / renderer / enumPrevious value: -[
- "auto",
- "lightpanda",
- "chrome",
- "playwright"
-]New value: +[
+ "auto",
+ "lightpanda",
+ "chrome",
+ "playwright",
+ "camoufox"
+]v0.16.0crw_check_crawl_status2 fields changedInput schema / properties / id / descriptionPrevious value: -"The crawl job ID returned by crw_crawl"New value: +"Crawl job id from crw_crawl"Input schema / properties / maxLengthAdded value: +{
+ "description": "Max chars per page content field; 0 = unbounded (default ~15000)",
+ "minimum": 0,
+ "type": "integer"
+}crw_crawl7 fields changedInput schema / properties / jsonSchema / descriptionPrevious value: -"JSON schema for LLM-based structured data extraction on each crawled page"New value: +"JSON schema for LLM extraction per page"Input schema / properties / maxDepth / descriptionPrevious value: -"Maximum crawl depth (default: 2)"New value: +"Max crawl depth (default 2)"Input schema / properties / maxPages / descriptionPrevious value: -"Maximum number of pages to crawl (default: 10)"New value: +"Max pages to crawl (default 10)"Input schema / properties / renderJs / descriptionPrevious value: -"Render JavaScript on every crawled page (true = force JS, false = HTTP only, omit = auto-detect or use the server's render_js_default)"New value: +"Force JS render (true), HTTP-only (false), omit = auto"Input schema / properties / renderer / descriptionPrevious value: -"Pin every crawled page to a specific renderer. \"auto\" (default if omitted) uses the configured fallback chain. Other values hard-pin with no fallback. Pinning a non-auto value implies renderJs:true unless renderJs:false is set explicitly."New value: +"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto)"Input schema / properties / url / descriptionPrevious value: -"The starting URL to crawl"New value: +"Starting URL"Input schema / properties / waitFor / descriptionPrevious value: -"Milliseconds to wait after JS rendering on each page"New value: +"Ms to wait after JS render per page"crw_map5 fields changedInput schema / properties / crawlFallback / descriptionPrevious value: -"If true (default), supplements sitemap discovery with a short BFS crawl when the sitemap returns enough URLs. Set false for sitemap-only mode (faster, may miss pages not in the sitemap)."New value: +"Supplement sitemap with a short BFS crawl (default true; false = sitemap-only)"Input schema / properties / limitAdded value: +{
+ "description": "Max URLs returned; 0 = unbounded (default 100)",
+ "minimum": 0,
+ "type": "integer"
+}Input schema / properties / maxDepth / descriptionPrevious value: -"Maximum crawl depth for discovery (default: 2)"New value: +"Max discovery depth (default 2)"Input schema / properties / url / descriptionPrevious value: -"The URL to map"New value: +"URL to map"Input schema / properties / useSitemap / descriptionPrevious value: -"Whether to use the site's sitemap.xml (default: true)"New value: +"Use sitemap.xml (default true)"crw_parse_file6 fields changedInput schema / properties / contentBase64 / descriptionPrevious value: -"Base64-encoded bytes of the PDF file"New value: +"Base64-encoded PDF bytes"Input schema / properties / filename / descriptionPrevious value: -"Original filename (optional; echoed in metadata.sourceFilename)"New value: +"Original filename (optional)"Input schema / properties / formats / descriptionPrevious value: -"Output formats (default: [\"markdown\"]). json/summary require a server LLM."New value: +"Output formats (default [\"markdown\"]); json/summary need a server LLM"Input schema / properties / jsonSchema / descriptionPrevious value: -"JSON schema for LLM-based structured extraction (when formats includes json)"New value: +"JSON schema for LLM extraction (when formats has json)"Input schema / properties / maxLengthAdded value: +{
+ "description": "Max chars per content field; 0 = unbounded (default ~15000)",
+ "minimum": 0,
+ "type": "integer"
+}Input schema / properties / parsers / descriptionPrevious value: -"Document parsers to apply (default: [\"pdf\"])"New value: +"Parsers to apply (default [\"pdf\"])"crw_scrape9 fields changedInput schema / properties / excludeTags / descriptionPrevious value: -"CSS selectors to exclude from output"New value: +"CSS selectors to exclude"Input schema / properties / formats / descriptionPrevious value: -"Output formats (default: [\"markdown\"])"New value: +"Output formats (default [\"markdown\"])"Input schema / properties / includeTags / descriptionPrevious value: -"CSS selectors to include (only content matching these selectors)"New value: +"CSS selectors to include"Input schema / properties / maxLengthAdded value: +{
+ "description": "Max chars per content field; 0 = unbounded (default ~15000)",
+ "minimum": 0,
+ "type": "integer"
+}Input schema / properties / onlyMainContent / descriptionPrevious value: -"Extract only the main content, removing nav/footer/etc (default: true)"New value: +"Strip nav/footer; main content only (default true)"Input schema / properties / renderJs / descriptionPrevious value: -"Render JavaScript before extracting (true = force JS, false = HTTP only, omit = auto-detect or use the server's render_js_default)"New value: +"Force JS render (true), HTTP-only (false), omit = auto"Input schema / properties / renderer / descriptionPrevious value: -"Pin this request to a specific renderer. \"auto\" (default if omitted) uses the configured fallback chain. Other values hard-pin to a single renderer with no fallback. Pinning a non-auto value implies renderJs:true unless renderJs:false is set explicitly."New value: +"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto)"Input schema / properties / url / descriptionPrevious value: -"The URL to scrape"New value: +"URL to scrape"Input schema / properties / waitFor / descriptionPrevious value: -"Milliseconds to wait after JS rendering for late content/XHRs"New value: +"Ms to wait after JS render for late content"crw_searchv0.15.2crw_parse_filecrw_search4 fields changedInput schema / properties / categories / descriptionPrevious value: -"Bias the search towards a category. `pdf` appends `filetype:pdf` to the query; `github`/`research` switch to topical engines."New value: +"Bias the search towards a category. Curated values: `pdf` appends `filetype:pdf` to the query; `github`/`research` switch to topical engines. Any other value (e.g. `science`, `it`, `news`, `files`) is passed straight through to SearXNG's native `categories` routing."Input schema / properties / categories / items / enumRemoved value: -[
- "github",
- "research",
- "pdf"
-]Input schema / properties / countryAdded value: +{
+ "description": "Country code for results (e.g. \"us\", \"tr\"). Hint to bias regional results; ignored if the underlying engine does not support it.",
+ "type": "string"
+}Output schema / (root)Previous value: -nullNew value: +{
+ "$defs": {
+ "searchResultItem": {
+ "properties": {
+ "category": {
+ "type": "string"
+ },
+ "description": {
+ "description": "Body snippet for the result. `snippet` is an alias of this field.",
+ "type": "string"
+ },
+ "position": {
+ "type": "integer"
+ },
+ "score": {
+ "type": "number"
+ },
+ "snippet": {
+ "description": "Alias of `description`. Always populated.",
+ "type": "string"
+ },
+ "title": {
+ "type": "string"
+ },
+ "url": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "url",
+ "title",
+ "description",
+ "snippet",
+ "position"
+ ],
+ "type": "object"
+ }
+ },
+ "properties": {
+ "data": {
+ "properties": {
+ "answer": {
+ "type": "string"
+ },
+ "citations": {
+ "type": "array"
+ },
+ "llmUsage": {
+ "type": "object"
+ },
+ "results": {
+ "oneOf": [
+ {
+ "items": {
+ "$ref": "#/$defs/searchResultItem"
+ },
+ "type": "array"
+ },
+ {
+ "properties": {
+ "images": {
+ "type": "array"
+ },
+ "news": {
+ "items": {
+ "$ref": "#/$defs/searchResultItem"
+ },
+ "type": "array"
+ },
+ "web": {
+ "items": {
+ "$ref": "#/$defs/searchResultItem"
+ },
+ "type": "array"
+ }
+ },
+ "type": "object"
+ }
+ ]
+ },
+ "warnings": {
+ "items": {
+ "type": "string"
+ },
+ "type": "array"
+ }
+ },
+ "required": [
+ "results"
+ ],
+ "type": "object"
+ },
+ "error": {
+ "type": "string"
+ },
+ "error_code": {
+ "type": "string"
+ },
+ "success": {
+ "type": "boolean"
+ },
+ "warning": {
+ "type": "string"
+ }
+ },
+ "required": [
+ "success",
+ "data"
+ ],
+ "type": "object"
+}Scored across 8 tools
Each tool targets a distinct operation: single-page scraping, full-site crawling, URL discovery, structured extraction, PDF parsing, and async job control. Status/cancel tools are clearly paired with their respective job types, so an agent can reliably select the right tool.
All tool names share the crw_ prefix and use a consistent lowercase snake_case style. Action verbs (crawl, scrape, extract, map, parse, check, cancel) are used predictably, making the API easy to navigate.
Eight tools is well-scoped for a crawling/extraction server. Each tool covers a distinct part of the workflow without redundancy or unnecessary surface area.
The core workflows are covered: crawling, scraping, extracting, mapping, parsing PDFs, and polling async jobs. The main gap is the lack of a cancel operation for crawl jobs, since extraction jobs have crw_cancel_extract but crawls have no equivalent.
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
Web scraping for AI agents. Extract text and metadata from any URL worldwide. $0.005/page.
Web tools for AI agents: scrape pages to Markdown, audit SEO, detect tech stacks, check sitemaps
Clean Markdown and AI-readability scoring for any URL. Built for AI agents.
