The crw-mcp server provides web scraping and crawling capabilities for AI agents via MCP tools:
crw_scrape: Scrape a single URL and return content as markdown, HTML, or links. Supports filtering by CSS selectors (include/exclude), extracting only main content, and LLM-based structured data extraction using a JSON schema.
crw_crawl: Start an async breadth-first crawl of a website with configurable depth and page limits, and optional structured data extraction per page. Returns a job ID for polling.
crw_check_crawl_status: Poll the status of an async crawl job and retrieve its results using the job ID.
crw_map: Discover all URLs on a website by crawling it and/or reading its sitemap.xml.
Additional capabilities include automatic JavaScript/SPA rendering via headless browsers (LightPanda or Chrome), and web search with full page content retrieval (cloud only).
curl -fsSL https://fastcrw.com/install | shRuns local and free, no account needed. To use the Cloud, paste your key into the same command and it installs the binary, connects the key, and registers the MCP server with the AI coding tools you already have:
curl -fsSL https://fastcrw.com/install | CRW_API_KEY=crw_live_... shcrw search "rust tutorials"Claude Code, Cursor, Codex, Gemini CLI, OpenCode and Windsurf are picked up
automatically when they are already set up; nothing else is touched, and your
key stays in ~/.config/crw/config.toml rather than being copied into each
tool. Add CRW_NO_AGENTS=1 to skip that step, or run crw setup on its own to
choose interactively.
1000 free credits, no credit card. Managed proxies, JS rendering and search, with nothing to run or keep up to date. Get my free key →
macOS and Linux, Intel and ARM. More install options →
Related MCP server: webpeel
Operation | Outcome |
Scrape | One URL to markdown, HTML, links, screenshots, or schema JSON |
Crawl | Follow a bounded site crawl and collect its pages |
Map | Discover URLs without scraping every page |
Search | Search the web and optionally scrape selected results |
Extract | Produce structured fields from one or many URLs |
On Firecrawl's own public 1,000-URL dataset, fastCRW recovered more truth than Crawl4AI and Firecrawl, matched the fastest median latency, and idled at ~14 MB RAM.
On a different benchmark entirely, answer accuracy rather than scrape recall, fastCRW answers 90.0% of the 600 AA-Omniscience questions correctly. Every product listed on the Artificial Analysis Search Index sits below it.
crw https://example.com # scrape, works right after install
crw search "rust async runtime" # search, after `crw setup`Using Cloud? Get an API key, then export it once:
export CRW_API_KEY="crw_live_..."
pip install crwfrom crw import CrwClient
client = CrwClient()
page = client.scrape("https://example.com", formats=["markdown"])
print(page["markdown"])npm install crw-sdkimport { CrwClient } from "crw-sdk";
const client = new CrwClient();
const page = await client.scrape("https://example.com", {
formats: ["markdown"],
});
console.log(page.markdown);Local mode and more SDK examples → · REST API →
npx -y crw-mcp@latest installInstalls the CRW skill and MCP server in your detected AI tools. crw setup can
also do this step, so either path is enough.
Manual setup →
Managed API | Local / self-hosted | |
Best for | Zero infrastructure and managed scaling | Data control, private networks, or custom infrastructure |
Start | Create an API key, then | Install and run |
Operations | Managed proxies, billing, and hosted capabilities | You choose renderers, search, auth, proxies, and capacity |
Capabilities and response shapes can differ by deployment:
/v1/capabilities · response shapes
The workspace requires Rust 1.85 or newer:
git clone https://github.com/us/crw
cd crw
make check-fastEngine and MCP server: AGPL-3.0. Python and TypeScript SDKs: MIT. Embedding license: hello@fastcrw.com.
Please respect website policies. Crawl and map follow robots.txt by default.
Request cancellation of an extract job. Returns the canonical status; cancelling remains non-terminal until the claimed URL settles.
| Name | Required | Description | Default |
|---|---|---|---|
| id | Yes | Extract job id from crw_extract |
| Name | Required | Description |
|---|---|---|
| id | Yes | |
| error | No | |
| status | Yes | |
| results | Yes | |
| success | Yes | |
| expiresAt | Yes | |
| tokensUsed | Yes | |
| creditsUsed | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Despite annotations already indicating destructive and non-read-only behavior, the description adds valuable context: cancellation is non-terminal until the claimed URL settles, and it returns canonical status. This goes beyond the structured annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tightly worded sentences. The first states the action; the second adds a critical caveat. No redundant information; front-loaded with purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a cancellation tool, the description covers purpose and the important async behavior. An output schema exists, so return values need not be described. Slightly more guidance on subsequent steps (e.g., checking status) could help, but it is not essential.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with the single 'id' parameter documented as 'Extract job id from crw_extract.' The description text itself adds no parameter details, so it relies entirely on the schema, which is adequate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states the action: 'Request cancellation of an extract job.' The verb 'cancel' and resource 'extract job' are specific, and it is distinct from sibling tools like crw_extract (create) and crw_check_extract_status (status check).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies use when you want to cancel an extract job, but it does not explicitly contrast with alternatives or mention when not to use it. No sibling tool is referenced, so guidance is implied rather than explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Poll an async crawl job and retrieve its pages.
| Name | Required | Description | Default |
|---|---|---|---|
| id | Yes | Crawl job id from crw_crawl | |
| maxLength | No | Max chars per page content field; 0 = unbounded (default ~15000) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint and idempotentHint, so the description adds only moderate behavioral context by confirming it polls and retrieves pages. The description does not disclose any additional traits beyond the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, concise sentence that is front-loaded and contains no unnecessary words. It efficiently conveys the tool's purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple polling tool with two parameters and no output schema, the description adequately explains the action (poll and retrieve pages). It could hint at the return format, but 'retrieve its pages' is sufficient for an agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, with clear parameter descriptions in the schema. The tool description adds no additional meaning beyond what is already in the schema, meeting the baseline for high coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool polls an async crawl job and retrieves its pages, providing a specific verb and resource. It distinguishes from siblings like crw_crawl (start) and crw_scrape (synchronous scrape).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage context by mentioning 'poll an async crawl job', indicating it's for checking ongoing crawls. However, it does not explicitly state when not to use or name alternatives, which is a minor gap.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Poll an extract job; returns status and, when complete, a per-URL results array.
| Name | Required | Description | Default |
|---|---|---|---|
| id | Yes | Extract job id from crw_extract |
| Name | Required | Description |
|---|---|---|
| id | Yes | |
| error | No | |
| status | Yes | |
| results | Yes | |
| success | Yes | |
| expiresAt | Yes | |
| tokensUsed | Yes | |
| creditsUsed | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations already indicate a safe, idempotent read operation. The description adds the behavioral detail that the tool returns status immediately and, upon completion, includes a per-URL results array, which is useful for understanding polling behavior. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence that directly states the action and outcome with no extraneous text. It earns a top score.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that an output schema exists and the tool has a single parameter, the description adequately covers the tool's purpose, return behavior, and usage context. It is complete for the tool's complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already defines 'id' as the extract job id from crw_extract, and the description does not add any additional parameter semantics. With 100% schema coverage, the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses the specific verb 'Poll' and identifies the resource as 'an extract job,' clearly distinguishing it from the similar sibling 'crw_check_crawl_status' by specifying 'extract' and the per-URL results array. It states exactly what the tool does and its output.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It clearly implies usage when checking the status of a previously submitted extract job, providing clear context. However, it does not explicitly mention alternatives or when not to use this tool, such as for crawl jobs, so it lacks explicit exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Start an async site crawl; returns a job id to poll with crw_check_crawl_status.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Starting URL | |
| waitFor | No | Ms to wait after JS render per page | |
| maxDepth | No | Max crawl depth (default 2) | |
| maxPages | No | Max pages to crawl (default 10) | |
| renderJs | No | Force JS render (true), HTTP-only (false), omit = auto | |
| renderer | No | Pin renderer; non-auto hard-pins and implies renderJs:true (default auto). 'camoufox' requires the server's opt-in camoufox tier to be configured. | |
| jsonSchema | No | Optional. A JSON Schema (draft 2020-12) describing fields to extract from each page via an LLM, e.g. {"type":"object","properties":{"title":{"type":"string"}}}. Free-form object. Omit to crawl without structured extraction. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare readOnlyHint=false, destructiveHint=false, idempotentHint=false, openWorldHint=true. The description adds the important behavioral trait that the crawl is async and returns a job id for polling, which informs the agent about the non-immediate nature and required follow-up. This goes beyond what annotations alone provide.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no superfluous text. The first sentence states the action and output, the second ties it to the sibling tool for follow-up. Every word earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite having 7 parameters and no output schema, the tool description covers the key context: async nature, job id return, and polling via a specific sibling. It does not explain the crawling scope or error handling, but the input schema fills in parameter details. For a start-crawl tool, this is fairly complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% (all 7 parameters have descriptions). The tool description does not add any additional parameter semantics beyond the schema, so baseline score of 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Start') and resource ('async site crawl'), clearly states the output (job id), and distinguishes from the sibling tool crw_check_crawl_status by indicating the polling workflow.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for usage: start a crawl then poll with the sibling tool. It does not explicitly list when to avoid using it or compare with other siblings like crw_map or crw_scrape, but the implication is strong enough for a straightforward tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Extract structured JSON from URLs via a prompt and/or JSON schema. Async job — poll crw_check_extract_status with the returned id. Needs an LLM.
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | URLs to extract from | |
| basis | No | Return per-field evidence: each top-level scalar property comes back with a source url, verbatim excerpt and honest status (supported/unverified/unsupported/notFound). Requires schema. | |
| prompt | No | Free-text extraction objective (required unless schema is given) | |
| schema | No | JSON Schema constraining the extracted output | |
| llmModel | No | ||
| llmApiKey | No | BYOK LLM API key | |
| llmProvider | No |
| Name | Required | Description |
|---|---|---|
| id | Yes | |
| urls | Yes | |
| status | Yes | |
| success | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate readOnlyHint=false and openWorldHint=true. The description adds behavioral nuance by disclosing the async job nature ('Async job — poll crw_check_extract_status with the returned id') and a key prerequisite ('Needs an LLM'). This provides context beyond the annotations, such as the non-blocking execution model and the requirement to track progress via a returned identifier.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is exactly two sentences: the first states the core purpose and method, the second covers the async workflow and prerequisite. Every clause adds value, and the most important information is front-loaded. There is no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 7 parameters, nested objects, and an output schema, the description efficiently covers the essential workflow (async, polling, LLM requirement). The existence of an output schema means return values are documented elsewhere. It does not discuss error handling or rate limits, but these are less critical given the asynchronous pattern and available schema documentation. Overall, it is complete enough for an agent to select and invoke the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema covers 71% of parameters with descriptions, so the baseline is 3. The description adds meaningful semantics by explaining that extraction works 'via a prompt and/or JSON schema', clarifying the relationship between prompt and schema parameters. It also highlights the LLM dependency, tying together llmModel/llmProvider/llmApiKey even though the schema doesn't explicitly state they are required. This goes beyond the schema's bare parameter list.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description begins with 'Extract structured JSON from URLs via a prompt and/or JSON schema', which clearly states the verb ('extract'), the resource ('URLs'), and the output format ('structured JSON'). It also distinguishes from sibling tools like crw_scrape (raw scraping) and crw_map by emphasizing structured extraction. The async note differentiates it from synchronous tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides context that it is an async job and must be polled via crw_check_extract_status, and that it needs an LLM. However, it does not explicitly state when to use this tool over alternatives such as crw_scrape or crw_crawl, nor any exclusions. The usage context is clear but lacks explicit when-to-use/when-not-to-use guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Discover URLs on a site via sitemap and/or a short crawl. Returns a URL list only, no page content.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to map | |
| limit | No | Max URLs to discover AND return; 0 = unbounded (default 100). Raise it (e.g. 50000) to pull deep/large sitemaps. | |
| maxDepth | No | Max discovery depth (default 2) | |
| useSitemap | No | Use sitemap.xml (default true) | |
| crawlFallback | No | Supplement sitemap with a short BFS crawl (default true; false = sitemap-only) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already provide readOnly, idempotent, and non-destructive hints. The description adds that output is URL list only, consistent with annotations, but no extra behavioral details beyond annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise, front-loaded sentences with no wasted words; every part adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity and rich schema+annotations, the description covers the main action and output. Minor gap: interaction of sitemap and crawl fallback is explained in schema, so description is complete enough.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has 100% parameter coverage with clear descriptions (e.g., limit 0 = unbounded, defaults). The description adds no further parameter meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool discovers URLs on a site via sitemap/crawl and explicitly says it returns only URLs, no content, distinguishing it from sibling crw_scrape and implying it's different from crw_crawl.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for URL discovery without content, but does not explicitly contrast with siblings like crw_crawl or provide when-to-use/not-use guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Parse a local PDF (base64 in contentBase64) to markdown. No OCR: scanned PDFs return empty markdown with a warning.
| Name | Required | Description | Default |
|---|---|---|---|
| formats | No | Output formats (default ["markdown"]); json/summary need a server LLM | |
| parsers | No | Parsers to apply (default ["pdf"]) | |
| filename | No | Original filename (optional) | |
| maxLength | No | Max chars per content field; 0 = unbounded (default ~15000) | |
| jsonSchema | No | Optional. A JSON Schema (draft 2020-12) describing fields to extract when formats includes "json", e.g. {"type":"object","properties":{"title":{"type":"string"}}}. Free-form object. | |
| contentBase64 | Yes | Base64-encoded PDF bytes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false, so the safety profile is clear. The description adds the behavioral trait that no OCR is performed and scanned PDFs return empty markdown with a warning, which is valuable context beyond annotations. It also clarifies that the input is base64-encoded local PDF bytes, but that's already in schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is one sentence, front-loaded with the main action ('Parse a local PDF...to markdown') and then a brief caveat. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 6 parameters and no output schema, the description covers the core functionality and a key limitation. However, it doesn't describe the response structure or that other output formats (json, summary) require a server LLM, though that's in the schema. The description is adequate for a simple parse tool but leaves some gap regarding output shape.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema provides descriptions for all 6 parameters (100% coverage), so the description doesn't need to add much. The description does reference contentBase64 and the markdown output, implicitly mapping to formats, but it doesn't explain the formats, jsonSchema, or maxLength parameters – though those are well-documented in the schema. Thus, the description adds minimal additional parameter semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool parses a local PDF (base64 in contentBase64) to markdown, which is a specific verb with resource and output format. It distinguishes itself from sibling tools by specifying 'local PDF' rather than URLs, aligning with crw_scrape/crw_crawl. The 'No OCR' caveat further defines its scope.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use it: when you have a local PDF as base64 and want markdown. It explicitly states a when-not scenario: scanned PDFs return empty markdown with a warning, which tells the agent to avoid using it for those. However, it doesn't name alternative tools for OCR or other formats, so it falls just short of explicit alternatives guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Scrape one URL to markdown, HTML, or links.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to scrape | |
| formats | No | Output formats (default ["markdown"]) | |
| waitFor | No | Ms to wait after JS render for late content | |
| renderJs | No | Force JS render (true), HTTP-only (false), omit = auto | |
| renderer | No | Pin renderer; non-auto hard-pins and implies renderJs:true (default auto). 'camoufox' requires the server's opt-in camoufox tier to be configured. | |
| maxLength | No | Max chars per content field; 0 = unbounded (default ~15000) | |
| excludeTags | No | CSS selectors to exclude | |
| includeTags | No | CSS selectors to include | |
| onlyMainContent | No | Strip nav/footer; main content only (default true) |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the agent knows this is a safe, non-destructive operation. The description adds minimal behavioral context beyond the 'one URL' scope, but does not disclose any additional behavior such as JS rendering defaults, pagination, or output limits. It does not contradict the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single 9-word sentence that is direct and front-loaded. Every word contributes to the core purpose. It is extremely concise without being verbose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (9 parameters, 4 output formats, multiple renderer options), the description is quite thin. The rich schema compensates for parameter details, but the description fails to mention the 'images' format and provides no context about renderer behavior or when to use optional parameters. For a simple scraper, it may be sufficient, but for an AI agent selecting among multiple similar tools, more context would be helpful.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema covers 100% of parameters with descriptions, so the baseline is 3. The description mentions markdown, HTML, and links, but omits 'images' from the formats enum, which is a minor gap. The description does not add meaning beyond the schema for the remaining parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (scrape), the resource (one URL), and the output formats (markdown, HTML, or links). The phrase 'one URL' distinguishes it from siblings like crw_crawl, which implies multi-page crawling. It is specific and unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies this tool is for scraping a single URL, since it says 'one URL'. However, it does not explicitly mention when to prefer this over crw_crawl, crw_map, or crw_extract, nor does it list any exclusions or prerequisites. Usage is implied rather than stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Each tool has a clearly distinct purpose: extract, scrape, crawl, map, parse file, and status checks are all differentiated by target resource and action. The async job polling and cancellation tools are specific to their respective job types, leaving no ambiguity.
All tools follow a consistent `crw_` prefix with snake_case and a verb_noun pattern (e.g., `crw_scrape`, `crw_check_extract_status`, `crw_parse_file`). The naming is predictable and uniform.
With 8 tools, the server is well-scoped for a web scraping/crawling toolkit. Each tool serves a distinct function, and the count fits comfortably within the ideal 3-15 range.
The toolkit covers core workflows: single-page scraping, site crawling, URL discovery, structured extraction, PDF parsing, and async job management. However, there is no cancellation for crawl jobs (asymmetric with extract cancellation), which is a minor gap in the crawl lifecycle.
Unclaimed servers have limited discoverability.
If you are the server author, to access and configure the admin panel.
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
Web scraping for AI agents. Extract text and metadata from any URL worldwide. $0.005/page.
Web scraping for AI agents. Converts URLs to clean, LLM-ready Markdown with anti-bot bypass.
Turn any URL into clean Markdown and structured data. Scrape, crawl, search and extract.
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/us/crw'
If you have feedback or need assistance with the MCP directory API, please join our Discord server
