crw-mcp
What's New
v0.4.0 (2026-04-22)
feat: add
crw-browse— interactive browser automation MCP server over CDPfeat: add SOCKS5 proxy support in
crw-rendererfeat: extract
crw-mcp-protocrate — shared JSON-RPC 2.0 types
fastCRW — Open Source Web Scraping API for AI Agents
Power AI agents with clean web data. Single Rust binary, zero config, Firecrawl-compatible API. The open-source Firecrawl alternative you can self-host for free — or use our managed cloud.
Don't want to self-host? Sign up free → — managed cloud with global proxy network, web search, and dashboard. Same API, zero infra. 500 free credits, no credit card required.
Related MCP server: webpeel
Why CRW? — Firecrawl & Crawl4AI Alternative
Single binary, 6 MB RAM — no Redis, no Node.js, no containers. Firecrawl needs 5 containers and 4 GB+. Crawl4AI requires Python + Playwright
5.5x faster than Firecrawl — 833ms avg vs 4,600ms (see benchmarks). P50 at 446ms
73/100 search win rate — beats Firecrawl (25/100) and Tavily (2/100) in head-to-head benchmarks
Free self-hosting — $0/1K scrapes vs Firecrawl's $0.83–5.33. No infra, no cold starts (85ms). No API key required for local mode
Agent ready — add to any MCP client in one command. Embedded mode: no server needed
Firecrawl-compatible API — drop-in replacement. Same
/v1/scrape,/v1/crawl,/v1/mapendpoints. HTML to markdown, structured data extraction, website crawler — all built-inBuilt for RAG pipelines — clean LLM-ready markdown output for vector databases and AI data ingestion
Open source — AGPL-3.0, developed transparently. Join our community
Metric | CRW (self-hosted) | fastcrw.com (cloud) | Firecrawl | Tavily | Crawl4AI |
Coverage (1K URLs) | 92.0% | 92.0% | 77.2% | — | — |
Avg Scrape Latency | 833ms | 833ms | 4,600ms | — | — |
Avg Search Latency | 880ms | 880ms | 954ms | 2,000ms | — |
Search Win Rate | 73/100 | 73/100 | 25/100 | 2/100 | — |
Idle RAM | 6.6 MB | 0 (managed) | ~500 MB+ | — (cloud) | — |
Cold start | 85 ms | 0 (always-on) | 30–60 s | — | — |
Self-hosting | Single binary | — | Multi-container | No | Python + Playwright |
Cost / 1K scrapes | $0 (self-hosted) | From $13/mo | $0.83–5.33 | — | $0 |
License | AGPL-3.0 | Managed | AGPL-3.0 | Proprietary | Apache-2.0 |
Web Scraping & Crawling Features
Core
Feature | Description |
Convert any URL to markdown, HTML, JSON, or links | |
Async BFS website crawler with rate limiting | |
Discover all URLs on a site instantly | |
Web search + content scraping (cloud) |
More
Feature | Description |
Send a JSON schema, get validated structured data back | |
Auto-detect SPAs, render via LightPanda or Chrome | |
Scrape any URL from your terminal — no server needed | |
Built-in stdio + HTTP transport for any AI agent |
Use Cases: RAG pipelines · AI agent web access · content monitoring · data extraction · HTML to markdown conversion · web archiving
Quick Start
# Install:
curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | CRW_BINARY=crw sh
# Scrape:
crw example.com
# Add to Claude Code (local):
claude mcp add crw -- npx crw-mcp
# Add to Claude Code (cloud — includes web search, 500 free credits at fastcrw.com):
claude mcp add -e CRW_API_URL=https://fastcrw.com/api -e CRW_API_KEY=your-key crw -- npx crw-mcpOr:
pip install crw(Python SDK) ·npx crw-mcp(zero install) ·brew install us/crw/crw(Homebrew) · All install options →
Scrape
Convert any URL to clean markdown, HTML, or structured JSON.
from crw import CrwClient
client = CrwClient(api_url="https://fastcrw.com/api", api_key="YOUR_API_KEY") # local: CrwClient()
result = client.scrape("https://example.com")
print(result["markdown"])Local mode:
CrwClient()with no arguments runs a self-contained scraping engine — no server, no API key, no setup. The SDK automatically downloads thecrw-mcpbinary on first use.
CLI:
crw example.com
crw example.com --format html
crw example.com --js --css 'article'Self-hosted (crw-server running on :3000):
curl -X POST http://localhost:3000/v1/scrape \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'Cloud:
curl -X POST https://fastcrw.com/api/v1/scrape \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'Output:
# Example Domain
This domain is for use in illustrative examples in documents.
You may use this domain in literature without prior coordination.Crawl
Scrape all pages of a website asynchronously.
from crw import CrwClient
client = CrwClient(api_url="https://fastcrw.com/api", api_key="YOUR_API_KEY") # local: CrwClient()
pages = client.crawl("https://docs.example.com", max_depth=2, max_pages=50)
for page in pages:
print(page["metadata"]["sourceURL"], page["markdown"][:80])# Start crawl
curl -X POST http://localhost:3000/v1/crawl \
-H "Content-Type: application/json" \
-d '{"url": "https://docs.example.com", "maxDepth": 2, "maxPages": 50}'
# Check status (use job ID from above)
curl http://localhost:3000/v1/crawl/JOB_IDMap
Discover all URLs on a site instantly.
from crw import CrwClient
client = CrwClient(api_url="https://fastcrw.com/api", api_key="YOUR_API_KEY") # local: CrwClient()
urls = client.map("https://example.com")
print(urls) # ["https://example.com", "https://example.com/about", ...]curl -X POST http://localhost:3000/v1/map \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'Search
Search the web and get full page content from results.
from crw import CrwClient
# Cloud only — requires fastcrw.com API key
client = CrwClient(api_url="https://fastcrw.com/api", api_key="YOUR_KEY")
results = client.search("open source web scraper 2026", limit=10)Cloud only:
search()requires a fastcrw.com API key (500 free credits, no credit card). Local/embedded mode providesscrape,crawl, andmap.
curl -X POST https://fastcrw.com/api/v1/search \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"query": "open source web scraper 2026", "limit": 10}'API Endpoints
Method | Endpoint | Description |
|
| Scrape a single URL, optionally with LLM extraction |
|
| Start async BFS crawl (returns job ID) |
|
| Check crawl status and retrieve results |
|
| Cancel a running crawl job |
|
| Discover all URLs on a site |
|
| Web search with optional content scraping (cloud only) |
|
| Health check (no auth required) |
|
| Streamable HTTP MCP transport |
Connect to AI Agents — MCP, Skill, Onboarding
Add CRW to any AI agent or MCP client in seconds.
Skill
Install the CRW skill to all detected agents with one command:
npx crw-mcp init --allRestart your agent after installing. Works with Claude Code, Cursor, Gemini CLI, Codex, OpenCode, and Windsurf.
MCP Server for AI Agents
Add CRW to any MCP-compatible client:
{
"mcpServers": {
"crw": {
"command": "npx",
"args": ["crw-mcp"]
}
}
}Works with Claude Desktop, Cursor, Windsurf, Cline, Continue.dev, and any MCP client.
Config file locations: Claude Code —
claude mcp add(no file edit). Claude Desktop —~/Library/Application Support/Claude/claude_desktop_config.json. Cursor —.cursor/mcp.json. Windsurf —~/.codeium/windsurf/mcp_config.json. All clients →
Cloud mode — adds crw_search for web search:
{
"mcpServers": {
"crw": {
"command": "npx",
"args": ["crw-mcp"],
"env": {
"CRW_API_URL": "https://fastcrw.com/api",
"CRW_API_KEY": "your-api-key"
}
}
}
}MCP Tools
Mode | Tool | Description |
Embedded + Cloud |
| Scrape a URL → markdown, HTML, JSON, links |
Embedded + Cloud |
| Start async BFS crawl (returns job ID) |
Embedded + Cloud |
| Poll crawl job status (use with |
Embedded + Cloud |
| Discover all URLs on a site |
Cloud only |
| Web search + optional content scraping |
Embedded mode (default): no server, no API key, no setup — the MCP binary runs a self-contained scraping engine. Cloud mode (
CRW_API_URLset): forwards calls to fastcrw.com, addscrw_search.
Claude Code shortcut:
# Local (embedded — no server, no API key):
claude mcp add crw -- npx crw-mcp
# Cloud (+ web search):
claude mcp add -e CRW_API_URL=https://fastcrw.com/api -e CRW_API_KEY=your-key crw -- npx crw-mcpListed on the MCP Registry
Agent Onboarding
Are you an AI agent? Fetch this skill file to configure CRW for the user's coding agents. The SKILL.md contains tool descriptions, usage examples, authentication setup, and edge case handling:
curl -s https://fastcrw.com/agent-onboarding/SKILL.mdBenchmark
Search — CRW vs Firecrawl vs Tavily (100 queries, concurrent)
Metric | CRW | Firecrawl | Tavily |
Avg Latency | 880ms | 954ms | 2,000ms |
Median Latency | 785ms | 932ms | 1,724ms |
Win Rate | 73/100 | 25/100 | 2/100 |
CRW is 2.3x faster than Tavily and won 73% of latency races. Full search benchmark →
Scrape — CRW vs Firecrawl (1,000 URLs, JS rendering enabled)
Tested on Firecrawl's scrape-content-dataset-v1:
Metric | CRW | Firecrawl v2.5 |
Coverage | 92.0% | 77.2% |
Avg Latency | 833ms | 4,600ms |
P50 Latency | 446ms | — |
Noise Rejection | 88.4% | noise 6.8% |
Idle RAM | 6.6 MB | ~500 MB+ |
Cost / 1K scrapes | $0 (self-hosted) | $0.83–5.33 |
Metric | CRW | Firecrawl |
Min RAM | ~7 MB | 4 GB |
Recommended RAM | ~64 MB (under load) | 8–16 GB |
Docker images | single ~8 MB binary | ~2–3 GB total |
Cold start | 85 ms | 30–60 seconds |
Containers needed | 1 (+optional sidecar) | 5 |
Run the benchmark yourself:
pip install datasets aiohttp
python bench/run_bench.pyInstall
MCP Server (crw-mcp) — recommended for AI agents
npx crw-mcp # zero install (npm)
pip install crw # Python SDK (auto-downloads binary)
brew install us/crw/crw-mcp # Homebrew
cargo install crw-mcp # Cargo
docker run -i ghcr.io/us/crw crw-mcp # DockerCLI (crw) — scrape URLs from your terminal
brew install us/crw/crw
# One-line install (auto-detects OS & arch):
curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | CRW_BINARY=crw sh
# APT (Debian/Ubuntu):
curl -fsSL https://apt.fastcrw.com/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/crw.gpg
echo "deb [signed-by=/usr/share/keyrings/crw.gpg] https://apt.fastcrw.com stable main" | sudo tee /etc/apt/sources.list.d/crw.list
sudo apt update && sudo apt install crw
cargo install crw-cliAPI Server (crw-server) — Firecrawl-compatible REST API
For serving multiple apps, other languages (Node.js, Go, Java), or as a shared microservice.
brew install us/crw/crw-server
# One-line install:
curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | CRW_BINARY=crw-server sh
# Docker:
docker run -p 3000:3000 ghcr.io/us/crwCustom port:
CRW_SERVER__PORT=8080 crw-server # env var
docker run -p 8080:8080 -e CRW_SERVER__PORT=8080 ghcr.io/us/crw # DockerWhen do you need
crw-server? Only if you want a REST API endpoint. The Python SDK (CrwClient()) and MCP binary (crw-mcp) both run a self-contained engine — no server required.
SDKs
Python
pip install crwfrom crw import CrwClient
# Cloud (fastcrw.com — includes web search):
client = CrwClient(api_url="https://fastcrw.com/api", api_key="YOUR_API_KEY")
# Local (embedded, no server needed):
# client = CrwClient()
# Scrape
result = client.scrape("https://example.com", formats=["markdown", "links"])
print(result["markdown"])
# Crawl (blocks until complete)
pages = client.crawl("https://docs.example.com", max_depth=2, max_pages=50)
# Map
urls = client.map("https://example.com")
# Search (cloud only)
results = client.search("AI news", limit=10, sources=["web", "news"])Requires: Python 3.9+. Local mode auto-downloads the
crw-mcpbinary on first use — no manual setup.
Community SDKs
crewai-crw— CRW scraping tools for CrewAI agentslangchain-crw— CRW document loader for LangChain
Node.js: No official SDK yet — use the REST API directly or
npx crw-mcpfor MCP. SDK examples →
Integrations
Frameworks: CrewAI · LangChain · Agno · Dify
Missing your favorite tool? Open an issue → · All integrations →
LLM Structured Extraction
Send a JSON schema, get validated structured data back using LLM function calling. Full extraction docs →
curl -X POST http://localhost:3000/v1/scrape \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/product",
"formats": ["json"],
"jsonSchema": {
"type": "object",
"properties": {
"name": { "type": "string" },
"price": { "type": "number" }
},
"required": ["name", "price"]
}
}'Configure the LLM provider:
[extraction.llm]
provider = "anthropic" # "anthropic" or "openai"
api_key = "sk-..." # or CRW_EXTRACTION__LLM__API_KEY env var
model = "claude-sonnet-4-20250514"JS Rendering
CRW auto-detects SPAs and renders them via a headless browser. Full JS rendering docs →
crw-server setup # downloads LightPanda, creates config.local.tomlRenderer | Protocol | Best for |
LightPanda | CDP over WebSocket | Low-resource environments (default); simple sites |
Chrome | CDP over WebSocket | Modern React/Vite/Next SPAs; recommended for production |
Playwright | CDP over WebSocket | Full browser compatibility |
Renderer choice matters for SPAs. LightPanda is fast and cheap but its JS runtime does not fully cover every modern bundle format. For React / Vite / Next sites whose content appears only after hydration, configure Chrome (or Playwright) alongside LightPanda — CRW will fall back to Chrome automatically when LightPanda returns a loading placeholder. Leaving LightPanda as the only renderer may silently return
"Loading..."-style shell content for these sites.
With Docker Compose, LightPanda runs as a sidecar automatically:
docker compose upCLI
Scrape any URL from your terminal — no server, no config. Full CLI docs →
crw example.com # markdown to stdout
crw example.com --format html # HTML output
crw example.com --format links # extract all links
crw example.com --js # with JS rendering
crw example.com --css 'article' # CSS selector
crw example.com --stealth # stealth mode (rotate UAs)
crw example.com -o page.md # write to fileSelf-Hosting
Once installed, start the server and optionally enable JS rendering:
crw-server # start REST API on :3000
crw-server setup # optional: downloads LightPanda for JS rendering
docker compose up # alternative: Docker with LightPanda sidecarSee the self-hosting guide for production hardening, auth, reverse proxy, and resource tuning.
Open Source vs Cloud
Self-hosted (free) | fastcrw.com Cloud | |
Core scraping | ✅ | ✅ |
JS rendering | ✅ (LightPanda/Chrome) | ✅ |
Web search | ❌ | ✅ |
Global proxy network | ❌ | ✅ |
Dashboard | ❌ | ✅ |
Commercial use without open-sourcing | Requires AGPL compliance | ✅ Included |
Cost | $0 | From $13/mo |
Sign up free → — 500 free credits, no credit card required.
Architecture
┌─────────────────────────────────────────────┐
│ crw-server │
│ Axum HTTP API + Auth + MCP │
├──────────┬──────────┬───────────────────────┤
│ crw-crawl│crw-extract│ crw-renderer │
│ BFS crawl│ HTML→MD │ HTTP + CDP(WS) │
│ robots │ LLM/JSON │ LightPanda/Chrome │
│ sitemap │ clean/read│ auto-detect SPA │
├──────────┴──────────┴───────────────────────┤
│ crw-core │
│ Types, Config, Errors │
└─────────────────────────────────────────────┘Crate | Description | |
Core types, config, and error handling | ||
HTTP + CDP browser rendering engine | ||
HTML → markdown/plaintext extraction | ||
Async BFS crawler with robots.txt & sitemap | ||
Axum API server (Firecrawl-compatible) | ||
MCP stdio server (embedded + proxy mode) | ||
Standalone CLI ( |
Configuration
Layered TOML config with environment variable overrides:
config.default.toml— built-in defaultsconfig.local.toml— local overrides (orCRW_CONFIG=myconfig)Environment variables —
CRW_prefix,__separator (e.g.CRW_SERVER__PORT=8080)
[server]
host = "0.0.0.0"
port = 3000
rate_limit_rps = 10
[renderer]
mode = "auto" # auto | lightpanda | playwright | chrome | none
[crawler]
max_concurrency = 10
requests_per_second = 10.0
respect_robots_txt = true
[auth]
# api_keys = ["fc-key-1234"]See full configuration reference.
Security
SSRF protection — blocks loopback, private IPs, cloud metadata (
169.254.x.x), IPv6 mapped addresses, and non-HTTP schemes (file://,data:)Auth — optional Bearer token with constant-time comparison
robots.txt — RFC 9309 compliant with wildcard patterns
Rate limiting — token-bucket algorithm, returns 429 with
error_codeResource limits — max body 1 MB, max crawl depth 10, max pages 1000
Resources
Contributing
Contributions are welcome! Please open an issue or submit a pull request.
Fork the repository
Install pre-commit hooks:
make hooksCreate your feature branch (
git checkout -b feat/my-feature)Commit your changes (
git commit -m 'feat: add my feature')Push to the branch (
git push origin feat/my-feature)Open a Pull Request
The pre-commit hook runs the same checks as CI (cargo fmt, cargo clippy, cargo test). Run manually with make check.
Contributors
License
CRW is open-source under AGPL-3.0. For a managed version without AGPL obligations, see fastcrw.com.
Get Started
Self-host free:
curl -fsSL https://raw.githubusercontent.com/us/crw/main/install.sh | sh— works in 30 secondsCloud: Sign up free → — 500 free credits, no credit card required
Questions? Join our Discord
It is the sole responsibility of end users to respect websites' policies when scraping. Users are advised to adhere to applicable privacy policies and terms of use. By default, CRW respects robots.txt directives.
Available Tools
8 toolscrw_cancel_extractCancel extract jobADestructiveIdempotentInspect
Request cancellation of an extract job. Returns the canonical status; cancelling remains non-terminal until the claimed URL settles.
| Name | Required | Description | Default |
|---|---|---|---|
| id | Yes | Extract job id from crw_extract |
Output Schema
| Name | Required | Description |
|---|---|---|
| id | Yes | |
| error | No | |
| status | Yes | |
| results | Yes | |
| success | Yes | |
| expiresAt | Yes | |
| tokensUsed | Yes | |
| creditsUsed | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Despite annotations already indicating destructive and non-read-only behavior, the description adds valuable context: cancellation is non-terminal until the claimed URL settles, and it returns canonical status. This goes beyond the structured annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tightly worded sentences. The first states the action; the second adds a critical caveat. No redundant information; front-loaded with purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a cancellation tool, the description covers purpose and the important async behavior. An output schema exists, so return values need not be described. Slightly more guidance on subsequent steps (e.g., checking status) could help, but it is not essential.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with the single 'id' parameter documented as 'Extract job id from crw_extract.' The description text itself adds no parameter details, so it relies entirely on the schema, which is adequate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states the action: 'Request cancellation of an extract job.' The verb 'cancel' and resource 'extract job' are specific, and it is distinct from sibling tools like crw_extract (create) and crw_check_extract_status (status check).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies use when you want to cancel an extract job, but it does not explicitly contrast with alternatives or mention when not to use it. No sibling tool is referenced, so guidance is implied rather than explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crw_check_crawl_statusCheck crawl statusARead-onlyIdempotentInspect
Poll an async crawl job and retrieve its pages.
| Name | Required | Description | Default |
|---|---|---|---|
| id | Yes | Crawl job id from crw_crawl | |
| maxLength | No | Max chars per page content field; 0 = unbounded (default ~15000) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint and idempotentHint, so the description adds only moderate behavioral context by confirming it polls and retrieves pages. The description does not disclose any additional traits beyond the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, concise sentence that is front-loaded and contains no unnecessary words. It efficiently conveys the tool's purpose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple polling tool with two parameters and no output schema, the description adequately explains the action (poll and retrieve pages). It could hint at the return format, but 'retrieve its pages' is sufficient for an agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, with clear parameter descriptions in the schema. The tool description adds no additional meaning beyond what is already in the schema, meeting the baseline for high coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool polls an async crawl job and retrieves its pages, providing a specific verb and resource. It distinguishes from siblings like crw_crawl (start) and crw_scrape (synchronous scrape).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage context by mentioning 'poll an async crawl job', indicating it's for checking ongoing crawls. However, it does not explicitly state when not to use or name alternatives, which is a minor gap.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crw_check_extract_statusCheck extract job statusARead-onlyIdempotentInspect
Poll an extract job; returns status and, when complete, a per-URL results array.
| Name | Required | Description | Default |
|---|---|---|---|
| id | Yes | Extract job id from crw_extract |
Output Schema
| Name | Required | Description |
|---|---|---|
| id | Yes | |
| error | No | |
| status | Yes | |
| results | Yes | |
| success | Yes | |
| expiresAt | Yes | |
| tokensUsed | Yes | |
| creditsUsed | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations already indicate a safe, idempotent read operation. The description adds the behavioral detail that the tool returns status immediately and, upon completion, includes a per-URL results array, which is useful for understanding polling behavior. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence that directly states the action and outcome with no extraneous text. It earns a top score.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that an output schema exists and the tool has a single parameter, the description adequately covers the tool's purpose, return behavior, and usage context. It is complete for the tool's complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema already defines 'id' as the extract job id from crw_extract, and the description does not add any additional parameter semantics. With 100% schema coverage, the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses the specific verb 'Poll' and identifies the resource as 'an extract job,' clearly distinguishing it from the similar sibling 'crw_check_crawl_status' by specifying 'extract' and the per-URL results array. It states exactly what the tool does and its output.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It clearly implies usage when checking the status of a previously submitted extract job, providing clear context. However, it does not explicitly mention alternatives or when not to use this tool, such as for crawl jobs, so it lacks explicit exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crw_crawlCrawl siteAInspect
Start an async site crawl; returns a job id to poll with crw_check_crawl_status.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Starting URL | |
| waitFor | No | Ms to wait after JS render per page | |
| maxDepth | No | Max crawl depth (default 2) | |
| maxPages | No | Max pages to crawl (default 10) | |
| renderJs | No | Force JS render (true), HTTP-only (false), omit = auto | |
| renderer | No | Pin renderer; browser tiers imply renderJs:true (default auto). 'camoufox' needs the opt-in tier configured. 'impersonated-http' is JS-less Chrome-TLS impersonation, never renderJs. | |
| jsonSchema | No | Optional. A JSON Schema (draft 2020-12) describing fields to extract from each page via an LLM, e.g. {"type":"object","properties":{"title":{"type":"string"}}}. Free-form object. Omit to crawl without structured extraction. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate openWorldHint=true (external side effects) and idempotentHint=false. The description adds the important async behavior and the job-id return contract, which are not in the annotations. It does not contradict any annotation and provides a clear behavioral expectation, though it omits details like potential load on target site (covered by openWorldHint).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, front-loaded sentence that states the core action and the immediate return value. There is zero fluff, and the most important behavioral fact (async + polling) is delivered first.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 7 parameters and no output schema, the description covers the essential contract: it starts an async crawl and returns a job id for polling. It does not explain the meaning of parameters (schema does) nor the external side effects (annotations do). The only minor gap is a lack of explicit warning about resource usage or cost, but openWorldHint and the async nature cover that adequately.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%: every parameter (url, waitFor, maxDepth, maxPages, renderJs, renderer, jsonSchema) has a descriptive definition. The tool description itself mentions none of these parameters, so it adds no additional semantics beyond the schema. Baseline 3 is appropriate because the schema already carries the full burden.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb ('start'), a clear resource ('site crawl'), and immediately explains the async nature and the follow-up mechanic (poll with crw_check_crawl_status). This differentiates it from siblings like crw_scrape or crw_check_crawl_status without ambiguity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It clearly establishes the typical workflow: start crawl, then poll status via the named sibling. It does not explicitly list alternatives or when not to use (e.g., for single-page extraction, use crw_scrape), but the async + polling context effectively guides usage and distinguishes it from crw_check_crawl_status.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crw_extractExtract structured dataAInspect
Extract structured JSON from URLs via a prompt and/or JSON schema. Async job — poll crw_check_extract_status with the returned id. Needs an LLM.
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | URLs to extract from | |
| basis | No | Return per-field evidence: each top-level scalar property comes back with a source url, verbatim excerpt and honest status (supported/unverified/unsupported/notFound). Requires schema. | |
| prompt | No | Free-text extraction objective (required unless schema is given) | |
| schema | No | JSON Schema constraining the extracted output | |
| llmModel | No | ||
| llmApiKey | No | BYOK LLM API key | |
| llmProvider | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| id | Yes | |
| urls | Yes | |
| status | Yes | |
| success | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate readOnlyHint=false and openWorldHint=true. The description adds behavioral nuance by disclosing the async job nature ('Async job — poll crw_check_extract_status with the returned id') and a key prerequisite ('Needs an LLM'). This provides context beyond the annotations, such as the non-blocking execution model and the requirement to track progress via a returned identifier.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is exactly two sentences: the first states the core purpose and method, the second covers the async workflow and prerequisite. Every clause adds value, and the most important information is front-loaded. There is no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 7 parameters, nested objects, and an output schema, the description efficiently covers the essential workflow (async, polling, LLM requirement). The existence of an output schema means return values are documented elsewhere. It does not discuss error handling or rate limits, but these are less critical given the asynchronous pattern and available schema documentation. Overall, it is complete enough for an agent to select and invoke the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema covers 71% of parameters with descriptions, so the baseline is 3. The description adds meaningful semantics by explaining that extraction works 'via a prompt and/or JSON schema', clarifying the relationship between prompt and schema parameters. It also highlights the LLM dependency, tying together llmModel/llmProvider/llmApiKey even though the schema doesn't explicitly state they are required. This goes beyond the schema's bare parameter list.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description begins with 'Extract structured JSON from URLs via a prompt and/or JSON schema', which clearly states the verb ('extract'), the resource ('URLs'), and the output format ('structured JSON'). It also distinguishes from sibling tools like crw_scrape (raw scraping) and crw_map by emphasizing structured extraction. The async note differentiates it from synchronous tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides context that it is an async job and must be polled via crw_check_extract_status, and that it needs an LLM. However, it does not explicitly state when to use this tool over alternatives such as crw_scrape or crw_crawl, nor any exclusions. The usage context is clear but lacks explicit when-to-use/when-not-to-use guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crw_mapMap site URLsARead-onlyIdempotentInspect
Discover URLs on a site via sitemap and/or a short crawl. Returns a URL list only, no page content.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to map | |
| limit | No | Max URLs to discover AND return; 0 = unbounded (default 100). Raise it (e.g. 50000) to pull deep/large sitemaps. | |
| maxDepth | No | Max discovery depth (default 2) | |
| useSitemap | No | Use sitemap.xml (default true) | |
| crawlFallback | No | Supplement sitemap with a short BFS crawl (default true; false = sitemap-only) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already provide readOnly, idempotent, and non-destructive hints. The description adds that output is URL list only, consistent with annotations, but no extra behavioral details beyond annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise, front-loaded sentences with no wasted words; every part adds value.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity and rich schema+annotations, the description covers the main action and output. Minor gap: interaction of sitemap and crawl fallback is explained in schema, so description is complete enough.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has 100% parameter coverage with clear descriptions (e.g., limit 0 = unbounded, defaults). The description adds no further parameter meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool discovers URLs on a site via sitemap/crawl and explicitly says it returns only URLs, no content, distinguishing it from sibling crw_scrape and implying it's different from crw_crawl.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for URL discovery without content, but does not explicitly contrast with siblings like crw_crawl or provide when-to-use/not-use guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crw_parse_fileParse PDFARead-onlyIdempotentInspect
Parse a local PDF (base64 in contentBase64) to markdown. No OCR: scanned PDFs return empty markdown with a warning.
| Name | Required | Description | Default |
|---|---|---|---|
| formats | No | Output formats (default ["markdown"]); json/summary need a server LLM | |
| parsers | No | Parsers to apply (default ["pdf"]) | |
| filename | No | Original filename (optional) | |
| maxLength | No | Max chars per content field; 0 = unbounded (default ~15000) | |
| jsonSchema | No | Optional. A JSON Schema (draft 2020-12) describing fields to extract when formats includes "json", e.g. {"type":"object","properties":{"title":{"type":"string"}}}. Free-form object. | |
| contentBase64 | Yes | Base64-encoded PDF bytes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false, so the safety profile is clear. The description adds the behavioral trait that no OCR is performed and scanned PDFs return empty markdown with a warning, which is valuable context beyond annotations. It also clarifies that the input is base64-encoded local PDF bytes, but that's already in schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is one sentence, front-loaded with the main action ('Parse a local PDF...to markdown') and then a brief caveat. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 6 parameters and no output schema, the description covers the core functionality and a key limitation. However, it doesn't describe the response structure or that other output formats (json, summary) require a server LLM, though that's in the schema. The description is adequate for a simple parse tool but leaves some gap regarding output shape.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema provides descriptions for all 6 parameters (100% coverage), so the description doesn't need to add much. The description does reference contentBase64 and the markdown output, implicitly mapping to formats, but it doesn't explain the formats, jsonSchema, or maxLength parameters – though those are well-documented in the schema. Thus, the description adds minimal additional parameter semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool parses a local PDF (base64 in contentBase64) to markdown, which is a specific verb with resource and output format. It distinguishes itself from sibling tools by specifying 'local PDF' rather than URLs, aligning with crw_scrape/crw_crawl. The 'No OCR' caveat further defines its scope.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use it: when you have a local PDF as base64 and want markdown. It explicitly states a when-not scenario: scanned PDFs return empty markdown with a warning, which tells the agent to avoid using it for those. However, it doesn't name alternative tools for OCR or other formats, so it falls just short of explicit alternatives guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crw_scrapeScrape URLARead-onlyIdempotentInspect
Scrape one URL to markdown, HTML, or links.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to scrape | |
| formats | No | Output formats (default ["markdown"]) | |
| waitFor | No | Ms to wait after JS render for late content | |
| renderJs | No | Force JS render (true), HTTP-only (false), omit = auto | |
| renderer | No | Pin renderer; browser tiers imply renderJs:true (default auto). 'camoufox' needs the opt-in tier configured. 'impersonated-http' is JS-less Chrome-TLS impersonation, never renderJs. | |
| maxLength | No | Max chars per content field; 0 = unbounded (default ~15000) | |
| excludeTags | No | CSS selectors to exclude | |
| includeTags | No | CSS selectors to include | |
| onlyMainContent | No | Strip nav/footer; main content only (default true) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, openWorldHint, idempotentHint, and destructiveHint=false, covering the safety profile. The description adds output format context but does not mention rendering behavior, potential delays, or response format. It does not contradict annotations and provides modest additional info about what the tool produces.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence that wastes no words and leads with the action and resource. It is appropriately concise for a tool with a well-documented schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has 9 parameters, only one required, and no output schema, the description is minimal. It lacks usage guidance and does not mention the 'images' format even though it is in the enum. However, the schema descriptions are thorough, so the agent can infer parameter behavior. The description is adequate but leaves gaps in when-to-use and output details.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so each parameter has a description. The tool description does not add parameter-specific meaning beyond that, and it does not clarify ambiguous terms. Baseline of 3 is appropriate given the schema fully documents the parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description 'Scrape one URL to markdown, HTML, or links' clearly identifies the verb (scrape), the resource (one URL), and the output types. It distinguishes from siblings like crw_crawl (multiple URLs) and crw_extract (structured data) by emphasizing a single URL and specific formats.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies single-page scraping but does not explicitly state when to prefer this over siblings like crw_crawl or crw_extract. There are no exclusions, alternatives, or conditions. The phrase 'one URL' hints at the scope, but no explicit guidance is provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
2 tool updates
v0.35.1- Changed
crw_crawl2 fields changed- changed
Input schema / properties / renderer / descriptionPrevious value: -"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto). 'camoufox' requires the server's opt-in camoufox tier to be configured."New value: +"Pin renderer; browser tiers imply renderJs:true (default auto). 'camoufox' needs the opt-in tier configured. 'impersonated-http' is JS-less Chrome-TLS impersonation, never renderJs." - changed
Input schema / properties / renderer / enumPrevious value: -[ - "auto", - "lightpanda", - "chrome", - "playwright", - "camoufox" -]New value: +[ + "auto", + "lightpanda", + "chrome", + "playwright", + "camoufox", + "impersonated-http" +]
- Changed
crw_scrape2 fields changed- changed
Input schema / properties / renderer / descriptionPrevious value: -"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto). 'camoufox' requires the server's opt-in camoufox tier to be configured."New value: +"Pin renderer; browser tiers imply renderJs:true (default auto). 'camoufox' needs the opt-in tier configured. 'impersonated-http' is JS-less Chrome-TLS impersonation, never renderJs." - changed
Input schema / properties / renderer / enumPrevious value: -[ - "auto", - "lightpanda", - "chrome", - "playwright", - "camoufox" -]New value: +[ + "auto", + "lightpanda", + "chrome", + "playwright", + "camoufox", + "impersonated-http" +]
5 tool updates
v0.30.0- Added
crw_cancel_extract - Changed
crw_check_extract_status1 field changed- changed
Output schema / (root)Previous value: -nullNew value: +{ + "additionalProperties": false, + "properties": { + "creditsUsed": { + "type": "integer" + }, + "error": { + "type": "string" + }, + "expiresAt": { + "format": "date-time", + "type": "string" + }, + "id": { + "type": "string" + }, + "results": { + "items": { + "additionalProperties": false, + "properties": { + "basis": { + "items": { + "type": "object" + }, + "type": "array" + }, + "basisWarnings": { + "items": { + "type": "object" + }, + "type": "array" + }, + "data": { + "additionalProperties": true, + "type": "object" + }, + "error": { + "type": "string" + }, + "llmInputHash": { + "type": "string" + }, + "llmUsage": { + "type": "object" + }, + "status": { + "enum": [ + "processing", + "completed", + "failed", + "cancelled" + ], + "type": "string" + }, + "url": { + "type": "string" + } + }, + "required": [ + "url", + "status" + ], + "type": "object" + }, + "type": "array" + }, + "status": { + "enum": [ + "processing", + "cancelling", + "completed", + "failed", + "cancelled" + ], + "type": "string" + }, + "success": { + "type": "boolean" + }, + "tokensUsed": { + "type": "integer" + } + }, + "required": [ + "success", + "id", + "status", + "results", + "expiresAt", + "tokensUsed" + ], + "type": "object" +}
- Changed
crw_extract1 field changed- changed
Output schema / (root)Previous value: -nullNew value: +{ + "additionalProperties": false, + "properties": { + "id": { + "type": "string" + }, + "status": { + "enum": [ + "processing" + ], + "type": "string" + }, + "success": { + "type": "boolean" + }, + "urls": { + "minimum": 0, + "type": "integer" + } + }, + "required": [ + "success", + "id", + "status", + "urls" + ], + "type": "object" +}
- Changed
crw_parse_file1 field changed- changed
Input schema / properties / formats / items / enumPrevious value: -[ - "markdown", - "plainText", - "links", - "json", - "summary" -]New value: +[ + "markdown", + "plainText", + "links", + "images", + "json", + "summary" +]
- Changed
crw_scrape1 field changed- changed
Input schema / properties / formats / items / enumPrevious value: -[ - "markdown", - "html", - "links" -]New value: +[ + "markdown", + "html", + "links", + "images" +]
1 tool update
v0.24.1- Changed
crw_extract1 field changed- added
Input schema / properties / basisAdded value: +{ + "description": "Return per-field evidence: each top-level scalar property comes back with a source url, verbatim excerpt and honest status (supported/unverified/unsupported/notFound). Requires schema.", + "type": "boolean" +}
2 tool updates
v0.22.0- Added
crw_check_extract_status - Added
crw_extract
1 tool update
v1.0.1- Changed
crw_map1 field changed- changed
Input schema / properties / limit / descriptionPrevious value: -"Max URLs returned; 0 = unbounded (default 100)"New value: +"Max URLs to discover AND return; 0 = unbounded (default 100). Raise it (e.g. 50000) to pull deep/large sitemaps."
3 tool updates
v0.18.0- Changed
crw_crawl4 fields changed- added
Input schema / properties / jsonSchema / additionalPropertiesAdded value: +true - changed
Input schema / properties / jsonSchema / descriptionPrevious value: -"JSON schema for LLM extraction per page"New value: +"Optional. A JSON Schema (draft 2020-12) describing fields to extract from each page via an LLM, e.g. {\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"}}}. Free-form object. Omit to crawl without structured extraction." - changed
Input schema / properties / renderer / descriptionPrevious value: -"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto)"New value: +"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto). 'camoufox' requires the server's opt-in camoufox tier to be configured." - changed
Input schema / properties / renderer / enumPrevious value: -[ - "auto", - "lightpanda", - "chrome", - "playwright" -]New value: +[ + "auto", + "lightpanda", + "chrome", + "playwright", + "camoufox" +]
- Changed
crw_parse_file2 fields changed- added
Input schema / properties / jsonSchema / additionalPropertiesAdded value: +true - changed
Input schema / properties / jsonSchema / descriptionPrevious value: -"JSON schema for LLM extraction (when formats has json)"New value: +"Optional. A JSON Schema (draft 2020-12) describing fields to extract when formats includes \"json\", e.g. {\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"}}}. Free-form object."
- Changed
crw_scrape2 fields changed- changed
Input schema / properties / renderer / descriptionPrevious value: -"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto)"New value: +"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto). 'camoufox' requires the server's opt-in camoufox tier to be configured." - changed
Input schema / properties / renderer / enumPrevious value: -[ - "auto", - "lightpanda", - "chrome", - "playwright" -]New value: +[ + "auto", + "lightpanda", + "chrome", + "playwright", + "camoufox" +]
6 tool updates
v0.16.0- Changed
crw_check_crawl_status2 fields changed- changed
Input schema / properties / id / descriptionPrevious value: -"The crawl job ID returned by crw_crawl"New value: +"Crawl job id from crw_crawl" - added
Input schema / properties / maxLengthAdded value: +{ + "description": "Max chars per page content field; 0 = unbounded (default ~15000)", + "minimum": 0, + "type": "integer" +}
- Changed
crw_crawl7 fields changed- changed
Input schema / properties / jsonSchema / descriptionPrevious value: -"JSON schema for LLM-based structured data extraction on each crawled page"New value: +"JSON schema for LLM extraction per page" - changed
Input schema / properties / maxDepth / descriptionPrevious value: -"Maximum crawl depth (default: 2)"New value: +"Max crawl depth (default 2)" - changed
Input schema / properties / maxPages / descriptionPrevious value: -"Maximum number of pages to crawl (default: 10)"New value: +"Max pages to crawl (default 10)" - changed
Input schema / properties / renderJs / descriptionPrevious value: -"Render JavaScript on every crawled page (true = force JS, false = HTTP only, omit = auto-detect or use the server's render_js_default)"New value: +"Force JS render (true), HTTP-only (false), omit = auto" - changed
Input schema / properties / renderer / descriptionPrevious value: -"Pin every crawled page to a specific renderer. \"auto\" (default if omitted) uses the configured fallback chain. Other values hard-pin with no fallback. Pinning a non-auto value implies renderJs:true unless renderJs:false is set explicitly."New value: +"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto)" - changed
Input schema / properties / url / descriptionPrevious value: -"The starting URL to crawl"New value: +"Starting URL" - changed
Input schema / properties / waitFor / descriptionPrevious value: -"Milliseconds to wait after JS rendering on each page"New value: +"Ms to wait after JS render per page"
- Changed
crw_map5 fields changed- changed
Input schema / properties / crawlFallback / descriptionPrevious value: -"If true (default), supplements sitemap discovery with a short BFS crawl when the sitemap returns enough URLs. Set false for sitemap-only mode (faster, may miss pages not in the sitemap)."New value: +"Supplement sitemap with a short BFS crawl (default true; false = sitemap-only)" - added
Input schema / properties / limitAdded value: +{ + "description": "Max URLs returned; 0 = unbounded (default 100)", + "minimum": 0, + "type": "integer" +} - changed
Input schema / properties / maxDepth / descriptionPrevious value: -"Maximum crawl depth for discovery (default: 2)"New value: +"Max discovery depth (default 2)" - changed
Input schema / properties / url / descriptionPrevious value: -"The URL to map"New value: +"URL to map" - changed
Input schema / properties / useSitemap / descriptionPrevious value: -"Whether to use the site's sitemap.xml (default: true)"New value: +"Use sitemap.xml (default true)"
- Changed
crw_parse_file6 fields changed- changed
Input schema / properties / contentBase64 / descriptionPrevious value: -"Base64-encoded bytes of the PDF file"New value: +"Base64-encoded PDF bytes" - changed
Input schema / properties / filename / descriptionPrevious value: -"Original filename (optional; echoed in metadata.sourceFilename)"New value: +"Original filename (optional)" - changed
Input schema / properties / formats / descriptionPrevious value: -"Output formats (default: [\"markdown\"]). json/summary require a server LLM."New value: +"Output formats (default [\"markdown\"]); json/summary need a server LLM" - changed
Input schema / properties / jsonSchema / descriptionPrevious value: -"JSON schema for LLM-based structured extraction (when formats includes json)"New value: +"JSON schema for LLM extraction (when formats has json)" - added
Input schema / properties / maxLengthAdded value: +{ + "description": "Max chars per content field; 0 = unbounded (default ~15000)", + "minimum": 0, + "type": "integer" +} - changed
Input schema / properties / parsers / descriptionPrevious value: -"Document parsers to apply (default: [\"pdf\"])"New value: +"Parsers to apply (default [\"pdf\"])"
- Changed
crw_scrape9 fields changed- changed
Input schema / properties / excludeTags / descriptionPrevious value: -"CSS selectors to exclude from output"New value: +"CSS selectors to exclude" - changed
Input schema / properties / formats / descriptionPrevious value: -"Output formats (default: [\"markdown\"])"New value: +"Output formats (default [\"markdown\"])" - changed
Input schema / properties / includeTags / descriptionPrevious value: -"CSS selectors to include (only content matching these selectors)"New value: +"CSS selectors to include" - added
Input schema / properties / maxLengthAdded value: +{ + "description": "Max chars per content field; 0 = unbounded (default ~15000)", + "minimum": 0, + "type": "integer" +} - changed
Input schema / properties / onlyMainContent / descriptionPrevious value: -"Extract only the main content, removing nav/footer/etc (default: true)"New value: +"Strip nav/footer; main content only (default true)" - changed
Input schema / properties / renderJs / descriptionPrevious value: -"Render JavaScript before extracting (true = force JS, false = HTTP only, omit = auto-detect or use the server's render_js_default)"New value: +"Force JS render (true), HTTP-only (false), omit = auto" - changed
Input schema / properties / renderer / descriptionPrevious value: -"Pin this request to a specific renderer. \"auto\" (default if omitted) uses the configured fallback chain. Other values hard-pin to a single renderer with no fallback. Pinning a non-auto value implies renderJs:true unless renderJs:false is set explicitly."New value: +"Pin renderer; non-auto hard-pins and implies renderJs:true (default auto)" - changed
Input schema / properties / url / descriptionPrevious value: -"The URL to scrape"New value: +"URL to scrape" - changed
Input schema / properties / waitFor / descriptionPrevious value: -"Milliseconds to wait after JS rendering for late content/XHRs"New value: +"Ms to wait after JS render for late content"
- Removed
crw_search
2 tool updates
v0.15.2- Added
crw_parse_file - Changed
crw_search4 fields changed- changed
Input schema / properties / categories / descriptionPrevious value: -"Bias the search towards a category. `pdf` appends `filetype:pdf` to the query; `github`/`research` switch to topical engines."New value: +"Bias the search towards a category. Curated values: `pdf` appends `filetype:pdf` to the query; `github`/`research` switch to topical engines. Any other value (e.g. `science`, `it`, `news`, `files`) is passed straight through to SearXNG's native `categories` routing." - removed
Input schema / properties / categories / items / enumRemoved value: -[ - "github", - "research", - "pdf" -] - added
Input schema / properties / countryAdded value: +{ + "description": "Country code for results (e.g. \"us\", \"tr\"). Hint to bias regional results; ignored if the underlying engine does not support it.", + "type": "string" +} - changed
Output schema / (root)Previous value: -nullNew value: +{ + "$defs": { + "searchResultItem": { + "properties": { + "category": { + "type": "string" + }, + "description": { + "description": "Body snippet for the result. `snippet` is an alias of this field.", + "type": "string" + }, + "position": { + "type": "integer" + }, + "score": { + "type": "number" + }, + "snippet": { + "description": "Alias of `description`. Always populated.", + "type": "string" + }, + "title": { + "type": "string" + }, + "url": { + "type": "string" + } + }, + "required": [ + "url", + "title", + "description", + "snippet", + "position" + ], + "type": "object" + } + }, + "properties": { + "data": { + "properties": { + "answer": { + "type": "string" + }, + "citations": { + "type": "array" + }, + "llmUsage": { + "type": "object" + }, + "results": { + "oneOf": [ + { + "items": { + "$ref": "#/$defs/searchResultItem" + }, + "type": "array" + }, + { + "properties": { + "images": { + "type": "array" + }, + "news": { + "items": { + "$ref": "#/$defs/searchResultItem" + }, + "type": "array" + }, + "web": { + "items": { + "$ref": "#/$defs/searchResultItem" + }, + "type": "array" + } + }, + "type": "object" + } + ] + }, + "warnings": { + "items": { + "type": "string" + }, + "type": "array" + } + }, + "required": [ + "results" + ], + "type": "object" + }, + "error": { + "type": "string" + }, + "error_code": { + "type": "string" + }, + "success": { + "type": "boolean" + }, + "warning": { + "type": "string" + } + }, + "required": [ + "success", + "data" + ], + "type": "object" +}
TDQS
Scored across 8 tools
Each tool targets a distinct operation: single-page scraping, full-site crawling, URL discovery, structured extraction, PDF parsing, and async job control. Status/cancel tools are clearly paired with their respective job types, so an agent can reliably select the right tool.
All tool names share the crw_ prefix and use a consistent lowercase snake_case style. Action verbs (crawl, scrape, extract, map, parse, check, cancel) are used predictably, making the API easy to navigate.
Eight tools is well-scoped for a crawling/extraction server. Each tool covers a distinct part of the workflow without redundancy or unnecessary surface area.
The core workflows are covered: crawling, scraping, extracting, mapping, parsing PDFs, and polling async jobs. The main gap is the lack of a cancel operation for crawl jobs, since extraction jobs have crw_cancel_extract but crawls have no equivalent.
Maintenance
Related MCP Connectors
Cloud scraping & crawling API for AI agents. Turn any URL into clean, LLM-ready markdown.
Web scraping for AI agents. Extract text and metadata from any URL worldwide. $0.005/page.
Web tools for AI agents: scrape pages to Markdown, audit SEO, detect tech stacks, check sitemaps
Clean Markdown and AI-readability scoring for any URL. Built for AI agents.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceWeb scraping MCP server for Al agents. 6 tools: extract clean text/markdown from any URL, structured scraping with CSS selectors, full-page screenshots via Playwright, link extraction with regex filtering, metadata extraction (OG tags, Twitter cards), and Google search. Free tier: 50 requests/IP/day.8MIT
- AlicenseNot gradedqualityBmaintenanceThe web data platform for AI agents. Fetch, search, crawl, extract, monitor, and screenshot any URL. 55+ domain extractors, 65-98% token savings. 7 MCP tools included.332 npm12AGPL 3.0
- AlicenseAqualityAmaintenanceWeb content extraction for AI agents. 10 tools: scrape, crawl, map, batch, extract, summarize, diff, brand, search, research. Uses TLS fingerprinting to bypass anti-bot without a headless browser. Outputs LLM-optimized markdown with 67% fewer tokens than raw HTML.102,346AGPL 3.0

HatFetchofficial
AlicenseAqualityAmaintenanceEnables LLM agents to read any website by scraping and crawling into clean Markdown, automatically bypassing bot detection with residential proxies.239 npmMIT