CrawlForge MCP Server
CrawlForge is an MCP server exposing 31 metered web-scraping, crawling, research and browser-automation tools callable directly from an AI assistant (Claude, Cursor, any MCP client).
Single-page reading —
scrapefetches one URL in multiple formats at once (markdown, html, rawHtml, text, links, metadata, branding, screenshots, LLM-structured JSON, or query-scoped highlights/question answers with verbatim offsets), with optional stealth escalation when a site blocks.Cheap targeted extraction —
fetch_url(raw API/JSON bodies),extract_text,extract_content,extract_links,extract_metadata,scrape_structured(CSS selectors),extract_embedded_state(Next.js/Nuxt/Redux JSON payloads).Site-wide discovery and crawling —
map_site(URL inventory, optional search ranking),crawl_deep(follow links across a site with depth/page limits, link-graph analysis),batch_scrape(2–50 URLs in one call),get_batch_results.Search and research —
search_web(up to 10 queries per call),reddit_search(posts, comments, full threads via archive),serp_rank(real Google organic position via DataForSEO),deep_research(multi-source report with conflict detection and source verification),agent(autonomous prompt-driven research with no URLs required).Browser automation —
scrape_with_actions(click, type, scroll, hover, select, press, navigate, JS, snapshots with@erefs) andbrowser_session(a persistent page keeping cookies and logins across calls).Anti-bot handling —
stealth_modewith Playwright or Camoufox engines, fingerprint randomisation and human-behaviour simulation.Content post-processing —
process_document(PDF/DOCX),summarize_content,analyze_content(sentiment, topics, entities, readability),generate_llms_txt,localization(geo/locale emulation),track_changes(change monitoring, scheduled and hosted monitors, alerts).LLM-powered extraction —
extract_with_llmandextract_structured, defaulting to a local Ollama model (no key, no data leaves the machine) with opt-in OpenAI/Anthropic, pluslist_ollama_models.Utility and controls —
read_result(search/slice/page results kept locally for 1 hour instead of refetching), PII redaction, robots.txt control, SSRF protection, credit metering with cost transparency, structured output and concurrency/pagination options.
Provides tools to scrape structured data from Amazon product pages, including details, reviews, and more.
Provides tools to scrape structured data from GitHub repositories, user profiles, and activity.
Provides web search capabilities using Google Search API, allowing retrieval of search results.
Provides tools to scrape structured data from npm package pages and registry.
Integrates with local Ollama models for LLM-powered content extraction without external API calls.
Offers integration with OpenAI models as an optional provider for LLM-powered extraction and analysis.
Provides tools to scrape structured data from Reddit posts, comments, and subreddits.
Provides tools to scrape structured data from YouTube videos, channels, and playlists.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@CrawlForge MCP Servercrawl docs.python.org and return the table of contents as markdown"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Table of Contents
Related MCP server: Crawl4AI MCP
🎯 Why CrawlForge?
31 MCP-native tools — scraping, crawling, search, real Google SERP rank tracking, deep research, an autonomous
agent, a unified multi-formatscrape, document processing, stealth browsing, stateful browser sessions, and more, callable directly from your AI assistant.Generous free tier — 1,000 credits to start instantly, no credit card. The grant is one-time rather than monthly, and the credits never expire.
Local-LLM by default —
extract_with_llmruns against a local Ollama model out of the box: no LLM API key, no per-token cost, and your data never leaves your machine. Cloud (OpenAI/Anthropic) is opt-in.LLM-ready output — clean Markdown, structured JSON (schema-driven), screenshots, links, and metadata from a single fetch.
Autonomous
agent— describe what you need in natural language; it plans, gathers, and shapes an answer under orchestrator-enforced hard stops (max steps/URLs/wall-clock) — no URLs required.Security-hardened — SSRF protection on every request, a fail-closed backend allow-list, a vetted action allowlist for browser automation, and per-tool credit gating.
Works everywhere MCP does — Claude Desktop, Claude Code, Cursor, and any other MCP-enabled client, configured in one command.
📊 CrawlForge vs. alternatives
CrawlForge MCP | Firecrawl | Raw scraping API | |
Native MCP server | ✅ 31 tools | ✅ | ❌ |
Free tier | ✅ 1,000 credits, one-time, never expire | Limited | Varies |
Self-hosted / local LLM extraction (Ollama) | ✅ default, $0/token | ❌ | ❌ |
Autonomous agent (no URLs needed) | ✅ | ✅ | ❌ |
Deep research with source verification | ✅ | Partial | ❌ |
Browser automation / actions | ✅ | ✅ | Varies |
Stealth / anti-detection engines | ✅ Chromium + Camoufox | ✅ | Add-on |
Pre-built site templates | ✅ 10 sites | ❌ | ❌ |
License | MIT | AGPL-3.0 | Proprietary |
Comparison reflects publicly documented capabilities at time of writing. CrawlForge is MIT-licensed and MCP-first — built to plug straight into AI coding assistants.
🚀 Quick Start (2 Minutes)
1. Install from NPM
npm install -g crawlforge-mcp-server2. Setup Your API Key (required)
Every tool requires a CrawlForge API key — new accounts get 1,000 free trial credits to start. The recommended path signs you in through the browser, so the key is never pasted into a terminal (a coding agent can run this for you and relay the URL):
crawlforge loginIt prints an approval URL; open it, approve, and the key is stored in ~/.crawlforge/config.json. Then run crawlforge init to register the MCP server with your client. Or use the interactive wizard, which also configures your clients:
npx crawlforge-setupThis will:
Guide you through getting your free API key
Configure your credentials securely
Auto-configure Claude Code and Cursor (if installed)
Verify your setup is working
Don't have an API key? Get one free at https://www.crawlforge.dev/signup
One-step setup (v4.6.0+):
crawlforge initdetects your API key, installs the agent skill, and idempotently merges the MCP config stanza into Claude Code, Claude Desktop, and Cursor. Usecrawlforge init --all --yesto configure every detected client non-interactively.
3. Configure Your IDE (if not auto-configured)
Add to claude_desktop_config.json:
{
"mcpServers": {
"crawlforge": {
"command": "npx",
"args": ["-y", "crawlforge-mcp-server"]
}
}
}Location:
macOS:
~/Library/Application Support/Claude/claude_desktop_config.jsonWindows:
%APPDATA%/Claude/claude_desktop_config.jsonLinux:
~/.config/Claude/claude_desktop_config.json
Restart Claude Desktop to activate.
The setup wizard automatically configures Claude Code by adding to ~/.claude.json:
{
"mcpServers": {
"crawlforge": {
"type": "stdio",
"command": "crawlforge-mcp"
}
}
}After setup, restart Claude Code to activate.
The setup wizard automatically configures Cursor by adding to ~/.cursor/mcp.json:
{
"mcpServers": {
"crawlforge": {
"type": "stdio",
"command": "crawlforge-mcp"
}
}
}Restart Cursor to activate.
n8n's built-in MCP Client Tool node connects over Streamable HTTP (works on n8n Cloud and self-hosted). Run the server in HTTP mode:
export CRAWLFORGE_API_KEY=your_api_key
npm run start:http # Streamable HTTP endpoint at http://localhost:10000/mcpThen point the MCP Client Tool node at http://<host>:10000/mcp with transport HTTP Streamable and a Bearer credential set to the same API key. On self-hosted n8n you can instead use the community n8n-nodes-mcp node over STDIO (npx -y crawlforge-mcp-server).
Full guide: docs/n8n-integration.md
Which launch command?
npx -y crawlforge-mcp-serverneeds no global install and always runs the published version (recommended for Claude Desktop). For a global install (npm i -g crawlforge-mcp-server), use the dedicatedcrawlforge-mcpbin — it resolves on yourPATH, so it survives Node/nvm version switches. The barecrawlforgecommand still launches the server when an MCP client spawns it over stdio (backward compatibility for configs created before v4.2.5); interactively it's the CLI — runcrawlforge mcpto start the server by hand.
📊 Available Tools
CrawlForge requires a CrawlForge API key — every tool is metered and consumes credits. New accounts get 1,000 free trial credits to start. Get a key at crawlforge.dev/signup.
All Tools (API key required)
Tool | Credits | What it does |
| 1 | Fetch content from any URL |
| 1 | Extract clean text from web pages |
| 1 | Get all links from a page |
| 1 | Extract page metadata (title, OG tags, schema.org) |
| 1 | Structured data from well-known sites (Amazon, GitHub, LinkedIn, YouTube, Reddit, Hacker News, npm, and more) without writing selectors |
| 1 | List the Ollama models installed locally (helps you pick a |
| 1 | Retrieve paginated results for a |
| 1 | Search, slice, read lines or a JSON path from a result a tool returned with |
| 2 | Unified single-fetch, multi-format extraction. Pass a |
| 2 | Extract structured data with CSS selectors |
| 2 | Read a page's embedded JavaScript state — |
| 2 | Enhanced content extraction |
| 2 | Discover and map website structure (optional |
| 2 | Multi-format document processing |
| 2 | Multi-language and geo-location management |
| 3 | Monitor content changes over time |
| 3 | Comprehensive content analysis |
| 3 | LLM-powered schema-driven extraction (your own LLM key or local Ollama) |
| 3 | Natural-language extraction. Defaults to a local Ollama model; pass |
| 3 | A browser page that stays open across calls, keeping its cookies and its login in between. |
| 4 | Generate intelligent summaries |
| 4 | Deep crawl entire websites |
| 5 | Search the web using Google Search API |
| 5 | Search Reddit posts/comments or read a full thread — reddit.com blocks direct scraping, so this reads the Arctic Shift community archive (free, no Reddit credentials). A Reddit-wide search spends a web search to discover posts, so it is priced with |
| 5 | Check where a domain ranks in Google's real organic SERP for a keyword (the position |
| 5 | Process multiple URLs simultaneously |
| 5 | Browser automation chains |
| 5 | Generate AI interaction guidelines |
| 5 | Anti-detection browser management |
| 8 | Autonomous research/extraction from a natural-language prompt — no URLs required. Plans, gathers, and shapes an answer under hard safety stops (max steps/URLs/wall-clock enforced by the orchestrator, never the LLM) |
| 10 | Multi-stage research with source verification |
Ten tools (scrape, fetch_url, extract_content, crawl_deep, batch_scrape, stealth_mode, scrape_with_actions, process_document, deep_research, extract_embedded_state) accept max_inline_chars (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS): a result over it comes back as a preview plus a result_handle for read_result, with the full result kept for 1 hour under ~/.crawlforge/results/ on your own machine — nothing is uploaded.
For the full canonical capabilities reference (all tools, CLI commands, stealth engines, research workflow), see SKILL.md.
💳 Pricing
Every tool is metered and requires an API key. New accounts get 1,000 free trial credits — no credit card required to start.
Plan | Credits | Best For |
Free | 1,000 one-time | Testing & personal projects |
Hobby ($19) | 5,000 / month | Small projects & development |
Professional ($99) | 50,000 / month | Professional use & production |
Business ($399) | 250,000 / month | Large scale operations |
All plans include:
Access to all 31 tools
Credits never expire; paid-plan credits roll over month to month
API access and webhook notifications
🔧 Advanced Configuration
Environment Variables
# Optional: Set API key via environment
export CRAWLFORGE_API_KEY="cf_live_your_api_key_here"
# Optional: Custom API endpoint (for enterprise)
export CRAWLFORGE_API_URL="https://api.crawlforge.dev"
# As of v3.0.18, this variable is validated against an allow-list of CrawlForge backend hosts.
# Optional: Local LLM (Ollama) overrides — extract_with_llm, extract_structured
# and deep_research all use Ollama when no cloud key is set
export OLLAMA_BASE_URL="http://localhost:11434" # default; set https://ollama.com for Ollama Cloud
export OLLAMA_DEFAULT_MODEL="gemma3:4b" # optional; unset = pick the best installed model automatically
# deep_research judges claims with gemma3:12b when it is installed (ollama pull gemma3:12b);
# conflict detection is on only with that model, or a cloud provider
export OLLAMA_EMBEDDING_MODEL="nomic-embed-text" # default: OLLAMA_DEFAULT_MODEL; used for semantic ranking in deep_research
export OLLAMA_API_KEY="..." # only for authenticated endpoints (required by Ollama Cloud; a local instance needs none)
export DISABLE_OLLAMA="true" # skip Ollama entirely and use CSS/keyword fallbacks
# Optional: Cloud LLM keys — only needed when you pass provider: "openai" or "anthropic"
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
# Optional: limit which tools this client sees — by name, by group, or both (comma-separated)
export CRAWLFORGE_TOOLS="scrape,search_web,extract_content"
export CRAWLFORGE_TOOL_GROUPS="basic,search,scrape" # unset = all tools; unknown names/groups are ignored with a warning
# Optional: deep_research stealth extraction fallback (v4.6.6) — see below
export RESEARCH_STEALTH_ENGINE="auto" # auto (default) | camoufox | chromium
export RESEARCH_STEALTH_FALLBACK="true" # set to "false" to disable entirely
export RESEARCH_MAX_STEALTH_RETRIES="8" # cap on stealth retries per research runMCP Spec Features
CrawlForge tracks the current MCP spec (2025-06-18) plus select experimental extensions:
Structured output —
scrape,map_site,serp_rank,reddit_search,search_web,extract_structured, andcrawl_deepreturn machine-parseablestructuredContentalongside the usual text, validated against a publishedoutputSchema; legacy clients keep working off the text.Self-correctable errors — invalid tool input now comes back as an
isError: trueresult the calling model can read and retry from, instead of a raw JSON-RPC protocol error.JSON Schema 2020-12 tool schemas, deterministic
tools/listordering (client prompt-cache friendly), and cacheable-result hints on read-only tools.Icons on the server, its tools, and its prompts.
Async tasks (experimental) on the four long-running tools —
crawl_deep,batch_scrape,deep_research,agent— for clients that support polling; synchronous results are still returned for clients that don't.
See docs/mcp-spec-adoption.md for wire-level examples and client-compatibility notes.
Local-LLM quickstart (extract_with_llm with Ollama)
extract_with_llm defaults to a local Ollama model — no LLM-provider key, no per-token LLM costs, and no data leaving your machine (the CrawlForge credit cost still applies).
# 1. Install Ollama: https://ollama.com
# 2. Pull any model from https://ollama.com/library
ollama pull llama3.2
# 3. Discover what's installed (from your MCP client)
# list_ollama_models()
# 4. Extract — defaults to Ollama with the model from step 2
# extract_with_llm({ url: "https://example.com", prompt: "…", model: "llama3.2" })Stealth extraction for deep_research (Camoufox)
deep_research automatically retries sources that block the normal fetch path (Reddit, Quora, forums, and Cloudflare/DataDome-protected pages return HTTP 403) through a real fingerprinted browser, then re-extracts from the rendered HTML. It's bounded (RESEARCH_MAX_STEALTH_RETRIES, default 8, plus a per-page timeout) and lazy — the browser stack only loads when a source is actually blocked.
Engine selection (RESEARCH_STEALTH_ENGINE):
auto(default) — prefer Camoufox (Firefox anti-detect), fall back to Chromium stealth, then plain fetch.camoufox— force Camoufox.chromium— force the Chromium stealth engine.
Headless Chromium cannot clear modern challenges (Cloudflare Turnstile, DataDome) — Camoufox can. In testing it recovered Quora and Trustpilot pages that were otherwise fully blocked. To enable it, install the optional dependency and run its one-time binary fetch:
# Camoufox is declared as an optional dependency, so a normal install already pulls it.
# If you installed with --no-optional, add it explicitly:
npm install camoufox
# One-time download of the Camoufox Firefox binary (~130 MB):
npx camoufox fetchWithout the Camoufox binary, deep_research silently falls back to Chromium stealth and then to plain fetch — no errors, just lower recovery on heavily-protected sites. Disable the whole fallback with RESEARCH_STEALTH_FALLBACK=false.
Note: Hard IP-reputation blocks (e.g. Reddit's edge
403) resist headless stealth from any IP and require residential/mobile proxies, which CrawlForge does not provide. See docs/stealth-engines.md for details.
Manual Configuration
Your configuration is stored at ~/.crawlforge/config.json:
{
"apiKey": "cf_live_...",
"userId": "user_...",
"email": "you@example.com"
}📖 Usage Examples
Once configured, use these tools in your AI assistant:
"Search for the latest AI news"
"Extract all links from example.com"
"Crawl the documentation site and summarize it"
"Monitor this page for changes"
"Extract product prices from this e-commerce site"🔒 Security & Privacy
Secure Authentication: API keys required for all metered tools
Local Storage: API keys stored securely at
~/.crawlforge/config.jsonHTTPS Only: All connections use encrypted HTTPS
No Data Retention: We don't store scraped data, only usage logs
Rate Limiting: Built-in protection against abuse
Compliance: Respects robots.txt and GDPR requirements
Security & Approvals
SSRF enforcement: Every scraped URL is validated before the request is sent — http/https only; blocks loopback, RFC1918, IPv6 private/link-local ranges, cloud metadata endpoints (GCP, Azure), and dangerous ports (SSH, SMTP, DNS, MySQL, Postgres, Redis, MongoDB, etc.). Redirects are re-validated each hop, capped at 5.
Backend endpoint guard (v3.0.18): The server's own calls to CrawlForge.dev use a separate fail-closed allow-list (
{crawlforge.dev, www.crawlforge.dev, api.crawlforge.dev}, HTTPS required). SettingCRAWLFORGE_API_URLto an arbitrary host is blocked at parse time.Action allowlist:
scrape_with_actionsaccepts only 11 action types (snapshot,wait,click,type,press,scroll,screenshot,executeJavaScript,select,hover,navigate). No download, file-write, or arbitrary cross-page navigation primitives exist —navigategoes through the same SSRF and robots.txt gate as the initial URL.JavaScript gate: The
executeJavaScriptaction throws by default. SetALLOW_JAVASCRIPT_EXECUTION=trueat deploy time to enable (not recommended in production).MCP Elicitation (v3.6.0): Four tools request user confirmation before executing expensive operations —
deep_research(>50 URLs),batch_scrape(sync mode, >25 URLs),crawl_deep(projected >500 pages),extract_structured(schema has >3 required fields with no LLM configured). Credit-low situations also elicit. Confirmation is best-effort: if the MCP client does not support elicitation the tool proceeds (fail-open).Per-tool credit gating: Every tool is wrapped with
withAuth()and is metered — credits are checked and deducted before execution, and a valid API key is required for every tool (fail-closed since v3.0.18).
See docs/sandboxing-and-approvals.md for the full reference.
Security Updates
v3.0.3 (2025-10-01): Removed authentication bypass vulnerability. All users must authenticate with valid API keys.
For the full security policy and how to report a vulnerability, see SECURITY.md.
🆘 Support
Documentation: https://www.crawlforge.dev/docs
Issues: GitHub Issues
Email: support@crawlforge.dev
Discord: Join our community
📄 License
MIT License - see LICENSE file for details.
🤝 Contributing
Contributions are welcome! Please read our Contributing Guide first.
Built with ❤️ by the CrawlForge team
Available Tools
31 toolsagentARead-only
Use this when you need an autonomous agent to research, navigate, and synthesise an answer from the web - no URLs required. The agent plans search queries, fetches and filters relevant pages, and returns a prose or structured answer. model:"pro" uses deep multi-source research. Hard limits: maxSteps<=10, maxUrls<=20, 120s wall-clock. Confirms before pro runs. Degraded-but-useful output if no LLM keys/Ollama. Not for a URL you already have (scrape) or a question one search answers (search_web). Cost: 8 credits, scales with maxUrls. Example: agent({prompt:"What are the top 5 MCP servers in 2025?", maxUrls:10})
| Name | Required | Description | Default |
|---|---|---|---|
| urls | No | Optional seed URLs to include (max 20) | |
| model | No | "default" = SamplingClient loop (no keys needed); "pro" = full ResearchOrchestrator | default |
| prompt | Yes | Natural-language task or question | |
| schema | No | Optional JSON schema for structured output | |
| maxUrls | No | Max URLs to fetch (hard cap: 20) | |
| maxSteps | No | Max fetch iterations (hard cap: 10) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and destructiveHint=false, and the description adds substantial behavioral context: it plans searches, fetches and filters pages, returns prose or structured answers, has hard limits, confirms before pro execution, degrades gracefully without LLM keys, and costs credits scaling with maxUrls. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Every sentence contributes: use case, mechanism, model variant, hard limits, pro confirmation, degraded mode, exclusions, cost, and example. The key trigger is front-loaded and the density is justified by the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 6-parameter autonomous agent tool with no output schema, the description provides enough context to call it correctly: expected return shape, limits, cost, failure behavior, alternatives, and a runnable example. Nothing essential seems missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema carries the baseline parameter documentation. The description adds meaning beyond the schema by explaining the maxUrls cost scaling, the model behavior difference, and a concrete invocation example that maps prompt and maxUrls together.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: an autonomous agent that researches, navigates, and synthesizes an answer from the web. It explicitly distinguishes itself from siblings by saying it is not for a URL you already have (scrape) or a question one search answers (search_web).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Opens with a direct when-to-use condition ('when you need an autonomous agent... no URLs required') and provides explicit exclusions for scrape and search_web. It also adds usage nuance for model selection, maxUrls costing, and the confirmation requirement before pro runs.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
analyze_contentARead-onlyIdempotent
Use this for NLP metrics on text you already hold - language detection, sentiment, topic extraction, entity recognition, readability score - for content auditing and classification. Takes text, not a URL. Not for reading a page (scrape returns the markdown to pass in). Cost: 3 credits. Example: analyze_content({text: "..article text..", options: {extractTopics: true, includeSentiment: true}})
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | The text content to analyze | |
| options | No | Analysis options |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already provide readOnly/idempotent/non-destructive hints, so the description's burden is lower. It adds useful behavioral context beyond annotations: the 3-credit cost and the constraint that it accepts inline text rather than a URL. It does not describe the output structure, but the listed metrics partially imply what will be returned.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but every sentence earns its place: when to use, what it analyzes, input format, exclusion, cost, and a concrete usage example. It is front-loaded with the core purpose and ends with the example, which is ideal.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a read-only, idempotent analysis tool, the description covers the key operational details: input expectations, cost, example invocation, and exclusionary context. The main gap is the lack of an explicit return-shape statement, but the absence of an output schema is partially mitigated by the listed analysis metrics.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, with text described as 'The text content to analyze'. The description adds concrete meaning by showing example option keys (extractTopics, includeSentiment) and clarifying that text is the raw content, not a URL. This compensates for the empty options object in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool analyzes text with NLP metrics (language detection, sentiment, topic extraction, entity recognition, readability) for content auditing and classification. It explicitly distinguishes itself from page-reading tools by saying 'Takes text, not a URL', separating it from scrape and related siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives explicit when-to-use guidance ('NLP metrics on text you already hold') and when-not-to-use guidance ('Not for reading a page'), even naming the exact alternative path: 'scrape returns the markdown to pass in'. This provides actionable routing to the agent.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
batch_scrapeARead-only
Use this to scrape 2-50 URLs in one call - product pages, news articles, competitor pages. Never loop scrape over a URL list. mode:"sync" returns results directly for up to ~25 URLs; mode:"async" with a webhook for larger batches, then get_batch_results. Not for one URL (scrape) or for discovering URLs (map_site). Cost: 5 credits. Example: batch_scrape({urls: ["https://a.com","https://b.com"], formats: ["json"], maxConcurrency: 5})
| Name | Required | Description | Default |
|---|---|---|---|
| mode | No | Processing mode: sync (wait) or async (background) | sync |
| urls | Yes | Array of URLs or URL objects to scrape | |
| formats | No | Output formats for scraped content | |
| webhook | No | Webhook configuration for async job notifications | |
| pageSize | No | Number of results per page | |
| jobOptions | No | Job management options for async processing | |
| redact_pii | No | Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off | |
| user_agent | No | Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with. | |
| includeFailed | No | Include failed URLs in results | |
| maxConcurrency | No | Maximum concurrent scraping requests | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. | |
| includeMetadata | No | Include page metadata in results | |
| extractionSchema | No | Schema for structured data extraction from each URL | |
| max_inline_chars | No | Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS) | |
| delayBetweenRequests | No | Delay in milliseconds between requests |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and destructiveHint=false, so the safety profile is known. The description adds value beyond that by disclosing cost (5 credits), the sync/async behavior difference, the webhook requirement for async, and the existence of get_batch_results. It doesn't contradict annotations. A small gap is that it doesn't describe return format or pagination, but that is secondary given the annotation coverage.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences plus an example. It front-loads the purpose, immediately states the key usage constraint (no looping), and provides a concrete example. Every sentence earns its place; no fluff or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex tool with 15 parameters, nested objects, and no output schema, the description covers the most critical decision points: when to use it, how to scale via modes, cost, and an example. It doesn't explain all parameters, but the schema does that. It also points to get_batch_results for async retrieval. Slight gap: it doesn't describe the sync return structure or pagination, but that is not explicitly required given the schema and the tool's nature.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so every parameter is described in the schema. The description adds an example that illustrates urls, formats, and maxConcurrency, and it explains the mode parameter's semantics (sync vs async) beyond the enum. However, it doesn't clarify other parameters like redact_pii, extractionSchema, or jobOptions, relying on the schema. This is adequate given the schema's thoroughness.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'scrape 2-50 URLs in one call' and names target content types. It immediately distinguishes itself from siblings by stating it is 'Not for one URL (scrape) or for discovering URLs (map_site)', so an agent can separate it from the many sibling scraping tools without opening schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives explicit when-to-use guidance: 'Never loop scrape over a URL list' and explains the mode choice (sync for up to ~25 URLs, async with webhook for larger batches). It names the alternatives (scrape, map_site) and the condition that selects them, plus an example call. This is comprehensive routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
browser_sessionA
Use this to drive a browser across several calls, keeping the page, its cookies and its login in between. The loop is: open a session on a URL, snapshot it to list the interactive elements with stable refs (@e1, @e2 ...), act on those refs, read the content, close. Because the page stays open you can look before each step instead of committing to a whole chain up front, so a wrong selector costs one call rather than all of them. Operations: open (url, stealth, ttl, activity_ttl, viewport), snapshot, act (the same action array as scrape_with_actions), read (formats), screenshot, close, list. Navigation invalidates refs, so snapshot again after one. robots.txt is respected on every navigation, and screenshots are stored as crawlforge://screenshot/{actionId} resources. A session expires 600s after it opens or 300s after its last use, whichever comes first, so close it when you are done. Not for a page that renders without interaction (scrape), and not for an interaction you can write out in advance - that is one scrape_with_actions call for 5. Cost: 3 credits to open; read 2; snapshot, act, screenshot, close and list 1 each. Example: browser_session({operation:"open", url:"https://app.com/login"}), then browser_session({operation:"snapshot", session_id:"..."})
| Name | Required | Description | Default |
|---|---|---|---|
| ttl | No | open: seconds the session may live at most (default 600) | |
| url | No | open: the URL to load the session on | |
| format | No | screenshot: image format | png |
| actions | No | act: the action array, same shape as scrape_with_actions. Target refs like "@e2" in `selector` | |
| formats | No | read: output formats | |
| quality | No | screenshot: JPEG quality | |
| stealth | No | open: run the session in the stealth browser | |
| timeout | No | Per-action timeout in ms | |
| selector | No | screenshot: capture just this element (a ref like "@e2" works) | |
| viewport | No | open: viewport size | |
| full_page | No | screenshot: capture the full scrollable page | |
| max_nodes | No | snapshot: cap on emitted nodes (default 200) | |
| operation | Yes | open a session, observe it, act on it, read it, or close it | |
| session_id | No | The id returned by operation:"open". Required by every operation except open and list | |
| activity_ttl | No | open: seconds the session may sit idle (default 300) | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. | |
| interactive_only | No | snapshot: only interactive elements get refs (false also emits headings and landmarks) | |
| max_inline_chars | No | Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS) | |
| continue_on_error | No | act: keep going past a failed action |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations cover only the safety profile (readOnlyHint=false, openWorldHint=true, idempotentHint=false). The description adds the operational traits an agent actually needs: session expiry (600s absolute / 300s idle), ref invalidation on navigation, robots.txt enforcement, screenshot resource URIs, and per-operation credit costs. This is well beyond what annotations provide.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with purpose, the operating loop, then rationale, alternatives, operations, constraints, and cost. Dense and mostly every sentence earns its place, though the operations enumeration partially duplicates schema descriptions and the end-of-paragraph cost/expiry detail could be tightened.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With 19 parameters, nested objects, no output schema, and seven sub-operations behind one entrypoint, this description supplies the missing glue: the workflow order, ref lifecycle, expiry semantics, credit costs, and a concrete call example. An agent can sequence a correct multi-call session from the description alone.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and each parameter already carries an operation prefix, so the baseline is 3. The description goes further by mapping operations to their parameter sets ('open (url, stealth, ttl, activity_ttl, viewport)') and by explaining ref semantics ('stable refs (@e1, @e2 ...)') and ref lifetime, which the schema cannot express. It does not document timeout, continue_on_error, or max_inline_chars behavior beyond what the schema says.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('drive a browser across several calls') and immediately establishes what makes it distinct: persistent state (page, cookies, login) across calls. It names the loop of operations and explicitly contrasts itself with scrape and scrape_with_actions, so an agent can identify it without opening the schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit when-to-use ('look before each step instead of committing to a whole chain up front, so a wrong selector costs one call rather than all of them') and explicit when-not-to-use ('Not for a page that renders without interaction (scrape), and not for an interaction you can write out in advance - that is one scrape_with_actions call for 5'). Names both alternatives and the selecting condition.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_deepARead-only
Use this to fetch many pages of one site by following links - a knowledge base, a docs index, a full-site audit. Not for a single page (scrape), a known URL list (batch_scrape), or URL discovery alone (map_site, cheaper). Cost: 4 credits base, grows with page count. Example: crawl_deep({url: "https://docs.example.com", max_depth: 3, max_pages: 200, extract_content: true})
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Starting URL for the crawl | |
| session | No | Shared cookie-jar/session for login-then-crawl workflows | |
| max_depth | No | Maximum crawl depth from starting URL | |
| max_pages | No | Maximum number of pages to crawl | |
| redact_pii | No | Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off | |
| concurrency | No | Number of concurrent requests | |
| domain_filter | No | Per-domain allow/deny lists and crawl rules | |
| respect_robots | No | Respect robots.txt directives | |
| extract_content | No | Extract page content during crawl | |
| follow_external | No | Follow links to external domains | |
| exclude_patterns | No | URL patterns to exclude (regex) | |
| include_patterns | No | URL patterns to include (regex) | |
| max_inline_chars | No | Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS) | |
| content_max_length | No | Maximum characters of page content to include per page (default 500); sets a truncated flag when trimmed | |
| enable_link_analysis | No | Compute PageRank/link-graph analysis over crawled pages | |
| import_filter_config | No | JSON string of a previously exported domain-filter config | |
| link_analysis_options | No | PageRank tuning options |
Output Schema
| Name | Required | Description |
|---|---|---|
| url | No | |
| view | No | Whether preview and read_result offsets index a text field or the pretty-printed JSON |
| _cost | No | Cost-transparency metadata (D3.5), present when injected into the text copy of the result |
| error | No | |
| stats | No | |
| cached | No | True when this response was replayed from an earlier crawl rather than crawled now; crawled_at gives its age |
| errors | No | |
| preview | No | The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON) |
| results | No | |
| session | No | |
| success | No | False only when the crawl was cancelled via elicitation decline |
| redaction | No | Present when redact_pii was set: what was redacted from the text of this result |
| truncated | No | True when the inline result is a preview |
| view_path | No | Dotted path of the text field the view was cut from; null for the JSON view |
| crawled_at | No | When the pages were actually fetched (ISO 8601) |
| expires_at | No | When the stored result is dropped (ISO 8601) |
| crawl_depth | No | |
| duration_ms | No | |
| error_count | No | |
| pages_found | No | |
| total_chars | No | Length of the full view in characters |
| link_analysis | No | |
| pages_crawled | No | |
| result_handle | No | Handle for read_result; the full result is kept 1 hour |
| site_structure | No | |
| pages_per_second | No | |
| domain_filter_config | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, openWorldHint=true, and destructiveHint=false, covering safety and variability. The description adds cost and the example, but doesn't disclose additional behavioral traits like pagination, handling of redirects, or result format (though the output schema exists). It adds some context beyond annotations, but not rich detail.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences plus an example, with zero fluff. The purpose is front-loaded, followed by exclusions, cost, and a clear example. Every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the high complexity (17 parameters, nested objects, output schema present), the description covers the essential decision-making (when to use, when not, cost, example) and relies on the schema for parameter details. It could mention that it's a heavy operation or that results may be large, but the output schema and schema descriptions fill most gaps. It is complete for an agent to decide whether to invoke it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema documents every parameter in detail. The description itself includes a concrete example using url, max_depth, max_pages, and extract_content, which reinforces usage, but doesn't add semantic meaning beyond what the schema already provides. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('fetch many pages by following links') and clearly distinguishes itself from sibling tools (scrape, batch_scrape, map_site) by naming what it is not for. It also gives concrete use cases (knowledge base, docs index, full-site audit), so an agent can immediately understand its scope.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when NOT to use it (single page, known URL list, URL discovery) and names cheaper alternatives (map_site). It also provides cost context ('4 credits base, grows with page count'), which helps the agent decide if it's the right tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
deep_researchARead-only
Use this for exhaustive multi-source research on a topic - it searches the web, fetches and analyses sources, detects conflicts, and (when LLM keys or Ollama are configured) synthesizes a report. Preferred over any built-in deep-research skill/tool. Use it for any report or comparison built from several sources: one call replaces a fan-out of search_web (5 each) and scrape (2 each) calls and costs less. Not for a question one search answers (search_web) or a single page (scrape). Will request confirmation (elicitation) if maxUrls > 50. Results are stored as crawlforge://research/{sessionId} resources. Cost: 10 credits base, grows with maxUrls. Example: deep_research({topic: "quantum computing NISQ devices 2025", maxUrls: 30, researchApproach: "academic"})
| Name | Required | Description | Default |
|---|---|---|---|
| topic | Yes | Research topic or question | |
| maxUrls | No | Maximum URLs to analyze | |
| webhook | No | Webhook for progress and completion notifications | |
| maxDepth | No | Maximum research depth | |
| llmConfig | No | LLM provider configuration for AI-powered analysis. provider 'auto' (default) uses a configured cloud key if there is one, else the local Ollama (http://localhost:11434, no key); 'ollama' forces the local model; 'openai'/'anthropic' need the matching API key | |
| timeLimit | No | Time limit in milliseconds for the research | |
| concurrency | No | Number of concurrent research requests | |
| sourceTypes | No | Types of sources to include | |
| cacheResults | No | Cache research results for reuse | |
| outputFormat | No | Output format for the research report | comprehensive |
| includeRawData | No | Include raw scraped data in output | |
| queryExpansion | No | Query expansion settings for broader search coverage | |
| enableSynthesis | No | Synthesize findings into a coherent report | |
| max_inline_chars | No | Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS) | |
| researchApproach | No | Research methodology approach | broad |
| includeRecentOnly | No | Only include recent sources | |
| includeActivityLog | No | Include detailed activity log | |
| credibilityThreshold | No | Minimum credibility score for sources (0-1) | |
| enableConflictDetection | No | Detect conflicting information across sources | |
| enableSourceVerification | No | Verify source credibility |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnly/openWorld annotations, it discloses that results are stored at crawlforge://research/{sessionId}, that maxUrls > 50 triggers a confirmation/elicitation step, that cost is 10 credits base and grows with maxUrls, and that synthesis depends on LLM keys or Ollama being configured. These details go well beyond what the annotations provide, and there is no contradiction.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but composed of only a few high-value sentences: purpose, routing, exclusions, behavioral notes, cost, and an example. Every sentence earns its place, and the core purpose is front-loaded before alternatives and cost details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 20-parameter tool with no output schema, it covers essential context: choice criteria, execution pipeline, confirmation behavior, cost, storage location, and LLM configuration dependency. It does not explicitly describe the report's return shape, but the schema's max_inline_chars documentation about preview plus result_handle partially fills that gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3; the description adds meaningful semantics around maxUrls (confirmation threshold and cost scaling) and includes a concrete example mapping topic, maxUrls, and researchApproach. Other parameters are left to the schema, but the schema already documents them thoroughly.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb-resource pair ('exhaustive multi-source research on a topic') and details the pipeline: web search, source fetch/analysis, conflict detection, and report synthesis when LLM/Ollama is configured. It also explicitly differentiates this tool from search_web and scrape, so an agent can distinguish it from relevant siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It states when to use it ('any report or comparison built from several sources'), names alternatives explicitly (search_web and scrape), and gives clear negative guidance ('Not for a question one search answers' or 'a single page'). It also declares it preferred over built-in deep-research skills, leaving no ambiguity.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_contentARead-onlyIdempotent
Use this for the readable body of an article-style page with ads, nav, footers and boilerplate removed - for RAG ingestion, summarisation, or LLM context. Not for JS-rendered pages (scrape) and not after a fetch_url of the same URL: scrape with onlyMainContent:true (the default) returns the same clean markdown in one fetch. Cost: 2 credits. Example: extract_content({url: "https://blog.example.com/post-title"})
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to extract content from | |
| options | No | Additional extraction options | |
| redact_pii | No | Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off | |
| user_agent | No | Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with. | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. | |
| max_inline_chars | No | Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and idempotentHint=true. The description adds valuable behavioral context beyond that: the 2-credit cost, the fact that it returns clean markdown, and the overlap with scrape's default behavior. It doesn't cover response format or pagination, but the safety profile is handled by annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Every sentence earns its place: purpose, exclusions with alternatives, cost, and example. The most critical information (what it is and when not to use it) is front-loaded. No filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 6 parameters and nested objects, the description covers the core decision (when to use), cost, and a minimal invocation example. Advanced parameters like options, redact_pii, and max_inline_chars are left to the schema, which documents them thoroughly. The description is complete enough for correct basic invocation and tool selection.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. The description includes a concrete example using the required url parameter, which reinforces its meaning, but it doesn't add semantic detail for the other parameters beyond what the schema already provides. The example is useful but not additive to the schema's own documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (extract) and resource (readable body of an article-style page with boilerplate removed). Distinguishes itself from siblings by explicitly naming scrape and fetch_url as alternatives for different scenarios, so an agent can tell them apart.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit when-to-use (RAG, summarization, LLM context) and when-not-to-use (JS-rendered pages, after fetch_url) with named alternatives. Also notes that scrape with onlyMainContent:true already returns the same clean markdown, eliminating redundant calls.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_embedded_stateARead-onlyIdempotent
Use this when a page's data lives in its embedded JavaScript state rather than its rendered HTML - Next.js (NEXT_DATA and React Server Component payloads), Nuxt, Apollo, Redux (INITIAL_STATE, PRELOADED_STATE), and blocks. One fetch, exact values, no LLM in the extraction path, so nothing can be fabricated. Payloads are routinely over a megabyte - pass path to return one subtree instead of the whole blob. Not for the rendered text of a page (scrape) or for sites built without a framework payload. Cost: 2 credits. Example: extract_embedded_state({url: "https://www.ticketmaster.com/discover/concerts", path: "next_data.props.pageProps"})
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to read embedded state from | |
| path | No | Return only this subtree instead of the whole payload. Dotted keys and array indexes, e.g. "next_data.props.pageProps" or "next_f[0].f" — not JSONPath (no wildcards, filters or recursion). State payloads are routinely over a megabyte; scope them. | |
| user_agent | No | Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with. | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. | |
| max_inline_chars | No | Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description adds meaningful behavioral context beyond annotations: 'One fetch, exact values, no LLM in the extraction path, so nothing can be fabricated,' and warns payloads are often over a megabyte. It also discloses the credit cost, which is useful operational behavior. No contradiction with the readOnly/idempotent/destructive annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Every sentence earns its place: purpose, framework guidance, behavioral guarantee, payload-size warning, non-usage exclusion, cost, and a realistic example. It is dense but well organized, front-loading the core decision before the details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite having no output schema, the description gives enough to invoke the tool correctly: when to use it, how to scope large results with path, an executable example, and clear non-targets. The schema covers parameter mechanics like robots and inline limits, so no critical guidance is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Input schema coverage is 100%, so the baseline is 3. The description adds value by explaining why path matters ('Payloads are routinely over a megabyte') and giving a concrete path example ('next_data.props.pageProps'), going beyond the schema's syntax note.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb and resource: extracting data from a page's embedded JavaScript state rather than rendered HTML. It explicitly identifies frameworks (Next.js, Nuxt, Apollo, Redux, application/json blocks) and distinguishes itself from scrape with 'Not for the rendered text of a page (scrape) or for sites built without a framework payload.'
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The opening line states the exact condition for use: 'Use this when a page's data lives in its embedded JavaScript state rather than its rendered HTML.' It also gives an explicit exclusion and the name of the alternative (scrape), plus an example showing the intended call shape.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_linksARead-onlyIdempotent
Use this to list the hyperlinks on one page - a crawl seed list, a broken-link audit, related resources. filter_external:true returns only outbound links. Not for a whole site (map_site), and not alongside a scrape of the same URL: scrape formats:["markdown","links"] returns both in one fetch. Cost: 1 credit. Example: extract_links({url: "https://example.com", filter_external: true})
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to extract links from | |
| base_url | No | Base URL for resolving relative links | |
| user_agent | No | Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with. | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. | |
| filter_external | No | Only return external links |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, and non-destructive behavior, so the safety profile is covered. The description adds value beyond annotations by disclosing the exact filter_external behavior, a cost of 1 credit, and the implicit single-page scope. It does not contradict any annotation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Every sentence earns its place: purpose, use cases, filter behavior, exclusions, cost, and an example. It is front-loaded with the primary purpose and routes around alternatives before ending with a concrete invocation. No filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-page link listing tool with no output schema, the description is complete: it names use cases, flags exclusions, gives the cost, and provides a copy-paste example. An agent has everything needed to call it correctly and to decide when it is the wrong choice.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all 5 parameters. The description reinforces the filter_external behavior and provides a concrete example, but adds no meaning beyond what the parameter descriptions already supply. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('list the hyperlinks on one page') and immediately enumerates concrete use cases (crawl seed list, broken-link audit, related resources). The description also distinguishes this tool from map_site and scrape, so an agent can select it confidently among many similar siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says when NOT to use it: not for a whole site (map_site), and not alongside a scrape of the same URL because scrape formats:['markdown','links'] returns both in one fetch. It also gives an example call, which is direct usage guidance. This is the clearest possible routing information.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_metadataARead-onlyIdempotent
Use this for a page's SEO metadata only: title, meta description, Open Graph tags, canonical URL, schema.org data. Not alongside a scrape of the same URL: scrape formats:["markdown","metadata"] returns both in one fetch. Cost: 1 credit. Example: extract_metadata({url: "https://example.com"})
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to extract metadata from | |
| user_agent | No | Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with. | |
| json_ld_types | No | Filter the returned JSON-LD to nodes of these schema.org types, e.g. ["Product","Offer"]. Subtypes match their parent: "Event" returns MusicEvent, "Offer" returns AggregateOffer, "ItemList" returns BreadcrumbList. Nodes are found at any depth, including inside @graph and nested inside a parent node. When set, json_ld carries only the matching nodes instead of the raw dump, and json_ld_type_counts reports how many matched per requested type. Documented types: ItemList, Product, Offer, Event, JobPosting, RealEstateListing — any other schema.org type is matched exactly. | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations already declare readOnly, idempotent, and non-destructive behavior, so the safety profile is covered. The description adds the cost detail ('1 credit') and a note about avoiding duplicate scrapes, but it does not disclose behavioral aspects like response shape, robots.txt handling, or user_agent defaults beyond what annotations already imply.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded: the core purpose appears first, followed by the anti-duplication warning, cost, and a usage example. Every sentence earns its place and there is no redundant wording.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's moderate complexity and the fully documented input schema, the description covers the essential operational details: purpose, when to avoid it, cost, and an example call. It does omit an explicit description of the response envelope, but the listed metadata fields already hint at the return content, so nothing critical is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all four parameters. The description only shows a URL example and does not go beyond the schema's parameter documentation, so it meets the baseline but provides no additional parameter-level meaning.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb and resource: extracting a page's SEO metadata, then enumerates exactly which fields are included (title, meta description, Open Graph tags, canonical URL, schema.org data). It also distinguishes itself from the scrape tool by explicitly warning against using it alongside a scrape of the same URL.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It states precisely when to use the tool ('for a page's SEO metadata only') and when not to use it ('Not alongside a scrape of the same URL'). It names the alternative and the exact condition that makes it better: scrape with formats ['markdown','metadata'] returns both in one fetch.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_structuredARead-onlyIdempotent
Use this to get a specific data shape from a page using a JSON schema when you can describe the fields but not their selectors - product details, job listings, event data. Uses an LLM by default; falls back to CSS selectors when no LLM is configured. Not for pages with stable markup whose selectors you know (scrape_structured, no LLM, cheaper). Cost: 3 credits. Example: extract_structured({url: "https://jobs.example.com/post/123", schema: {properties: {title: {type:"string"}, salary: {type:"string"}}, required:["title"]}})
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to extract structured data from | |
| prompt | No | Natural language instructions for extraction | |
| schema | Yes | JSON schema defining the data structure to extract | |
| llmConfig | No | LLM provider configuration for AI-powered extraction | |
| user_agent | No | Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with. | |
| selectorHints | No | CSS selector hints to guide extraction | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. | |
| verify_numbers | No | Numeric provenance guard (default: true): every price or numeric value the LLM returns must appear literally in the page source, else it is returned as null with a reason in `provenance.unverified`. Set false to get the model's raw numbers back, including ones it derived (a count, a sum, a total) rather than read off the page. | |
| fallbackToSelectors | No | Fall back to CSS selector extraction if LLM is unavailable |
Output Schema
| Name | Required | Description |
|---|---|---|
| url | No | |
| data | No | Extracted fields matching the requested schema |
| _cost | No | Cost-transparency metadata (D3.5), present when injected into the text copy of the result |
| error | No | |
| success | No | False when the extraction errored, or a required field came back missing, empty, or in the wrong shape |
| confidence | No | |
| provenance | No | |
| validation | No | |
| schema_used | No | |
| processingTime | No | |
| extractionNotes | No | |
| extraction_method | No | "llm" | "css_fallback" | "keyword_fallback" | "none" |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations already cover readOnly, idempotent, and non-destructive traits. The description adds genuinely useful behavioral disclosure beyond those: 'Uses an LLM by default; falls back to CSS selectors when no LLM is configured' and 'Cost: 3 credits.' There is no contradiction with the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with a clear purpose and uses only three sentences plus an example. Every sentence earns its place: the when-to-use rule, a priority alternative, the cost, and a complete invocation example. Nothing is redundant or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present, the description does not need to explain return values. It fully covers how to select this tool versus alternatives, how to invoke it (example), the cost, the default LLM behavior, and its fallback path. Combined with the 100% schema coverage, this is a complete definition.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and the example in the description demonstrates the `url` and `schema` parameters in context. However, the description mostly restates the schema's own parameter descriptions and does not add deeper explanation for parameters like `llmConfig`, `selectorHints`, or `verify_numbers` beyond what the input schema already provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description's first sentence states exactly what the tool does: 'Use this to get a specific data shape from a page using a JSON schema.' It gives the precise condition 'when you can describe the fields but not their selectors' and cites concrete use cases (product details, job listings, event data), which clearly distinguishes it from the sibling scrape_structured.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly sets when to use it ('when you can describe the fields but not their selectors') and when not to use it ('Not for pages with stable markup whose selectors you know') and directly names the alternative: scrape_structured. It also notes the default LLM use and the fallback path, so an agent is not left to guess.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_textARead-onlyIdempotent
Use this for a page's plain text or markdown with tags, scripts and styles removed - the cheapest read of a static HTML page. Use output_format:"markdown" for RAG. Not for article pages (extract_content strips nav and boilerplate), JS-rendered pages (scrape), or when you also want links or metadata (scrape with several formats, one fetch). Cost: 1 credit. Example: extract_text({url: "https://example.com/article", output_format:"markdown"})
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to extract text from | |
| redact_pii | No | Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off | |
| user_agent | No | Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with. | |
| output_format | No | Output format: "text" (default) or "markdown" — use markdown for RAG workflows | text |
| remove_styles | No | Remove style tags before extraction | |
| remove_scripts | No | Remove script tags before extraction | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, and non-destructive behavior, so the bar is lower. The description adds useful behavioral context: it strips tags/scripts/styles, targets static HTML rather than JS-rendered pages, costs 1 credit, and this is the cheapest read. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences with no waste: usage, exclusions, cost, and an example are all packed efficiently. The most important scoping information is front-loaded, and every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a read-only extraction tool with 7 parameters fully described in the schema and no output schema, the description covers everything needed to select and invoke it correctly: when to use it, alternatives, cost, format choice, and a concrete example. Nothing essential is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3; the schema already documents each parameter thoroughly. The description adds marginal value by pointing out the markdown format for RAG and showing an example call, but it does not provide substantial meaning beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States exactly what the tool returns: a page's plain text or markdown with tags, scripts, and styles removed, and identifies it as the cheapest read of a static HTML page. It also names sibling tools (extract_content, scrape) and the conditions that select them, so the agent can distinguish it from alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly says when to use it (static HTML plain text/markdown, cheap read, RAG with markdown) and when not to (article pages, JS-rendered pages, or when links/metadata are needed), naming the alternative for each exclusion. The example call also gives concrete usage guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_with_llmARead-only
Extract data from a URL or text using a natural-language prompt. Defaults to a local Ollama model (http://localhost:11434, no API key required) and picks an installed model itself; pass model to choose one, or provider: "openai" or "anthropic" with the matching API key to use a cloud model instead. Not for known selectors (scrape_structured) or a schema-shaped result (extract_structured). Call list_ollama_models only when a model name is rejected. Cost: 3 credits plus your LLM provider's own charge.
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | URL to fetch and extract from (one of url/content required) | |
| model | No | Override the model. For ollama, pass a name returned by list_ollama_models (e.g. 'llama3.2', 'qwen2.5:7b'). Defaults: openai='gpt-4o-mini', anthropic='claude-haiku-4-5-20251001', ollama='llama3.2' or $OLLAMA_DEFAULT_MODEL. | |
| prompt | Yes | Natural-language extraction instruction | |
| schema | No | Optional JSON-schema for output shape (used as Ollama structured-outputs format when provider is 'ollama') | |
| content | No | Pre-fetched text to extract from (one of url/content required) | |
| provider | No | LLM provider. Defaults to 'ollama' (local, no key, http://localhost:11434). Use 'openai' or 'anthropic' for cloud models (requires the matching API key). | auto |
| maxTokens | No | Maximum output tokens | |
| user_agent | No | Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with. | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. | |
| verify_numbers | No | Numeric provenance guard (default: true): every price or numeric value the LLM returns must appear literally in the page source, else it is returned as null with a reason in `provenance.unverified`. Set false to get the model's raw numbers back, including ones it derived (a count, a sum, a total) rather than read off the page. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and destructiveHint=false, covering the safety profile. The description adds valuable behavioral context beyond annotations: default Ollama local endpoint with no API key, automatic model selection, cloud providers needing matching API keys, and cost (3 credits plus the provider's own charge). No contradiction with annotations; the extra context justifies a score above baseline.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is information-dense but every sentence earns its place: purpose, default behavior, model/provider overrides, exclusions, when to call a sibling, and cost. The core purpose is front-loaded, and there is no fluff or restatement of the tool name.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 10 parameters and no output schema, the description covers the key invocation decisions: required prompt, url/content mutual requirement, provider/model defaults, and when to use alternatives. It omits the return-value shape, but the absence of an output schema makes this a minor gap; the essential behavioral and routing information is present.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema documents every parameter. The description adds meaning beyond the schema by specifying default model names per provider, explaining that 'auto' resolves to Ollama, and clarifying that url/content are mutually required. This is genuinely additive to the structured field descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: 'Extract data from a URL or text using a natural-language prompt.' It explicitly names the siblings it is not for ('Not for known selectors (scrape_structured) or a schema-shaped result (extract_structured)'), so an agent can distinguish it from extract_structured and scrape_structured without opening their schemas.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit when-to-use context: use for natural-language extraction from URL/text, not for selector-based scraping or schema-shaped results, and call list_ollama_models only when a model name is rejected. Provider selection (ollama vs openai/anthropic) and API key requirements are also made clear, leaving little to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
fetch_urlARead-onlyIdempotent
Use this for a raw HTTP body - JSON, XML, plain text, an API response - or for the status code, headers and response time. Returns the body unprocessed. Not for HTML you intend to read: scrape returns markdown from one fetch, so fetch_url followed by extract_* is a double fetch. Not for JS-rendered or bot-protected pages (scrape, then stealth_mode). Supports custom headers (e.g. auth tokens) and a timeout. Cost: 1 credit. Example: fetch_url({url: "https://api.example.com/v1/items", timeout: 15000})
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to fetch content from | |
| headers | No | Custom HTTP headers to include in the request | |
| timeout | No | Request timeout in milliseconds (1000-30000) | |
| user_agent | No | Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with. | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. | |
| max_inline_chars | No | Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false, so the safety profile is covered. The description adds significant behavioral context beyond that: it discloses the cost (1 credit), the timeout and custom headers support, the unprocessed return of the body, and the specific behavior of respect_robots (recording false setting against API key and returning a warning). It also explains max_inline_chars behavior (preview + result_handle). No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences, front-loaded with the core purpose and immediately contrasting with siblings. Every sentence earns its place: purpose, exclusions, features, cost, and an example. No fluff, no repetition of schema details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 6 parameters and no output schema, the description covers all necessary decision points: what to use it for, what to avoid, what it returns, cost, and edge cases (large results). It also names the sibling tools an agent might consider. The absence of an output schema is mitigated because the description explicitly states it returns the body and also mentions status, headers, and response time. The example shows a minimal valid call. Everything an agent needs to invoke correctly is present.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description adds value by hinting at typical header usage ('e.g. auth tokens') and by including an example that demonstrates timeout usage. It also clarifies the implication of respect_robots and max_inline_chars in context, going slightly beyond the schema descriptions. However, the schema already fully documents each parameter, so the increment is modest.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a clear verb-resource pair ('fetch a URL') and immediately distinguishes itself from siblings by specifying exactly what it returns (raw body, status, headers, response time) and what it does not (HTML rendering). It names the sibling 'scrape' and the extract_* family as alternatives, making differentiation explicit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit when-to-use guidance (raw HTTP body, API responses) and when-not-to-use (HTML for reading, JS-rendered or bot-protected pages), and points to alternatives: 'scrape returns markdown from one fetch' and 'scrape, then stealth_mode'. It also includes a concrete example call, reinforcing correct invocation.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
generate_llms_txtARead-onlyIdempotent
Use this to generate an llms.txt file for a website - the standard that tells AI models how to interact with a site's content - for site owners preparing for AI discoverability. Not for reading a site's existing llms.txt (fetch_url on /llms.txt). Cost: 5 credits. Example: generate_llms_txt({url: "https://example.com"})
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The website URL to generate llms.txt for | |
| format | No | Output format: llms.txt, llms-full.txt, or both | both |
| outputOptions | No | Output customization and organization details | |
| analysisOptions | No | Website analysis options for depth, scope, and detection | |
| complianceLevel | No | Compliance level for generated guidelines | standard |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is covered. The description adds useful behavioral context: the cost (5 credits) and a concrete example call. It doesn't describe side effects or rate limits, but the annotations cover the key behavioral traits. The cost disclosure is a valuable addition beyond annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences with zero waste. The core purpose is front-loaded, the exclusion is stated early, and the cost plus example are packed efficiently. Every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 5 parameters, 100% schema coverage, and no output schema, the description covers the essential context: what it does, when to use it, cost, and an example. The only minor gap is that it doesn't describe what the return value looks like, but with no output schema and rich parameter schema, the description is largely complete for an agent to select and invoke the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all parameters. The description adds a concrete example with the url parameter, which is helpful, but doesn't add meaning beyond the schema for the other parameters. Baseline 3 is appropriate when the schema does the heavy lifting.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('generate'), a specific resource ('llms.txt file for a website'), and the purpose ('standard that tells AI models how to interact with a site's content'). It also explicitly distinguishes itself from reading an existing llms.txt by pointing to fetch_url on /llms.txt. This is a clear, specific purpose that differentiates it from siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says when to use this tool ('for site owners preparing for AI discoverability') and when not to use it ('Not for reading a site's existing llms.txt (fetch_url on /llms.txt)'). It names the alternative tool and the condition that selects it. This is explicit usage guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_batch_resultsARead-onlyIdempotent
Retrieve paginated results for a batch_scrape job by the batchId it returned. Not a scraping tool - it re-reads an already-paid batch. Poll only async jobs; a sync batch has already returned its results. Cost: 1 credit. Example: get_batch_results({batchId: "batch_1234567890_abc", page: 2, pageSize: 25})
| Name | Required | Description | Default |
|---|---|---|---|
| page | No | Page number (1-based) | |
| batchId | Yes | The batch ID returned by batch_scrape | |
| pageSize | No | Number of results per page | |
| max_inline_chars | No | Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already cover read-only, idempotent, and non-destructive behavior. The description adds valuable behavioral context beyond annotations: the batch must be already-paid, polling is only for async jobs, and there is a per-call credit cost. It does not describe the return envelope, but the schema's max_inline_chars description partially covers the preview/handle behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded: it states the core purpose, key constraints, cost, and a concrete example in just three sentences. There is no filler or repetition of schema content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple paginated read tool with full schema coverage and supportive annotations, the description covers the essential operational details: how to identify the batch, when to poll, cost, and a working example. The large-result fallback is documented in the schema, so nothing critical is missing for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description adds value with a concrete invocation example showing batchId, page, and pageSize in context, reinforcing the relationship between the batchId and the batch_scrape call. It does not add detail for max_inline_chars, but the schema already documents it fully.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states a specific verb ('Retrieve'), a specific resource ('paginated results for a batch_scrape job'), and the key identifier ('batchId'). It also explicitly distinguishes itself from a scraping tool, which sets it apart from siblings like batch_scrape and scrape.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives explicit when-to-use guidance: poll only async jobs, and don't use for sync batches whose results were already returned. It also clarifies it re-reads an already-paid batch and notes the cost, leaving no ambiguity about when this tool applies.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_ollama_modelsARead-onlyIdempotent
List the Ollama models installed locally, to choose a model value for extract_with_llm. Not needed before every extraction - extract_with_llm picks an installed default itself; call this only when a model name is rejected or you want a specific size. Requires Ollama running on http://localhost:11434 (or $OLLAMA_BASE_URL). Cost: 1 credit.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, and non-destructive, so the bar for additional behavior is lower. The description adds meaningful context: the Ollama prerequisite (localhost:11434 or $OLLAMA_BASE_URL), the associated credit cost, and the 'installed locally' scope. It doesn't cover error behavior, but that is a minor gap for a simple read-only list.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, each earning its place: the core operation, the usage guidance, and the prerequisite/cost. The key information is front-loaded and there is no wasted text.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter, read-only list tool, the description covers purpose, when to use, the alternative, the infrastructure prerequisite, and cost. The output is implied by the purpose ('list... to choose a model'), so nothing needed for correct invocation is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are zero parameters, so the baseline is 4. The description correctly avoids inventing parameter guidance and instead clarifies the output's intended use ('model' value for extract_with_llm), which adds value beyond the empty schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('List'), a precise resource ('Ollama models installed locally'), and a clear purpose (choosing a model for extract_with_llm). It is distinct from all sibling tools and leaves no ambiguity about what the tool returns.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly says the tool is not needed before every extraction, names the alternative behavior (extract_with_llm picks a default), and gives two concrete conditions for calling it: a rejected model name or a need for a specific size. This is excellent when-to-use guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
localizationA
Use this to scrape geo-restricted content or emulate a specific locale/timezone - region-specific pricing, geo-blocks, searching in another language. Use operation:"configure_country" to set country context for the scraping calls that follow. Not for an ordinary page read (scrape). Cost: 2 credits. Example: localization({operation:"configure_country", countryCode:"DE", language:"de"})
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | URL — required for handle_geo_blocking; for auto_detect it is optional metadata used only as a TLD country hint (the page is never fetched) | |
| content | No | Page content (HTML or plain text) to analyze — required for auto_detect; no fetching is performed | |
| currency | No | ISO 4217 currency code (e.g. 'USD', 'EUR') | |
| language | No | Language code (e.g. 'en', 'fr', 'de') | |
| response | No | HTTP response for geo-blocking analysis | |
| timezone | No | IANA timezone identifier (e.g. 'America/New_York') | |
| operation | No | Localization operation to perform | configure_country |
| userAgent | No | Custom user agent string | |
| countryCode | No | ISO 3166-1 alpha-2 country code | |
| geoLocation | No | GPS coordinates for geolocation emulation | |
| searchParams | No | Search parameters for localized search queries | |
| customHeaders | No | Custom HTTP headers for localized requests | |
| proxySettings | No | Proxy configuration for geo-targeted requests | |
| acceptLanguage | No | Accept-Language header value | |
| browserOptions | No | Browser context options for locale emulation |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark the tool as non-read-only, non-idempotent, and non-destructive. The description adds a stateful side effect (configure_country sets country context for subsequent scraping calls), clarifies that it is not a normal page fetch, and discloses the 2-credit cost. This adds useful context beyond the annotations, though it stops short of describing effects of the other operations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three compact sentences plus one illustrative example. Purpose is front-loaded, cost and exclusion are included without bloat. The opening clause is slightly run-on, but the overall length and structure are appropriate.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description fully supports the configure_country path and excludes ordinary scraping, but the tool exposes eight operations and has no output schema. The other operations such as localize_search, auto_detect, handle_geo_blocking, and get_stats are not mapped to use cases or expected outputs, which is a clear gap for a multi-operation tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3. The description adds value with a concrete invocation binding operation, countryCode, and language, and explains what configure_country does with those parameters. The rest of the parameters are already well documented in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States concrete use cases: scraping geo-restricted content, emulating a locale/timezone, region-specific pricing, geo-blocks, and localized search. It also distinguishes itself from the 'scrape' sibling with 'Not for an ordinary page read (scrape)' and provides a concrete example so an agent can tell what the tool is for.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit when-to-use conditions (geo-restricted content, locale/timezone emulation) and an explicit when-not ('Not for an ordinary page read (scrape)'). The example shows exactly how to invoke the primary configuration operation. It does not map out all eight operations, but the tool-vs-alternative decision is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
map_siteARead-onlyIdempotent
Use this to list a site's URLs without fetching page bodies - reads sitemap.xml when available, otherwise follows links. Not for page content (scrape, or crawl_deep for many pages) and not for the links on one page (extract_links). Cost: 2 credits. Example: map_site({url: "https://example.com", include_sitemap: true, max_urls: 500})
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The website URL to map | |
| search | No | When set, rank discovered URLs by relevance to this string and emit ranked_urls:[{url,score}] | |
| max_urls | No | Maximum number of URLs to discover | |
| user_agent | No | Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with. | |
| domain_filter | No | Per-domain allow/deny lists and URL include/exclude patterns | |
| group_by_path | No | Group URLs by path segments | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. | |
| include_sitemap | No | Include sitemap.xml data in results | |
| include_metadata | No | Include page metadata for each URL | |
| import_filter_config | No | JSON string of a previously exported domain-filter config |
Output Schema
| Name | Required | Description |
|---|---|---|
| urls | No | Flat array of URLs, or grouped-by-path object when group_by_path=true (default) |
| _cost | No | Cost-transparency metadata (D3.5), present when injected into the text copy of the result |
| base_url | No | |
| metadata | No | Per-URL metadata when include_metadata=true |
| site_map | No | |
| statistics | No | |
| total_urls | No | |
| ranked_urls | No | Present only when the `search` param was set |
| filter_stats | No | |
| domain_filter_config | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, openWorldHint=true, idempotentHint=true, and destructiveHint=false, covering the safety profile. The description adds behavioral detail beyond annotations: it 'reads sitemap.xml when available, otherwise follows links' and notes the 2-credit cost, which helps set expectations about how the tool operates and what it costs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the core purpose, followed by sibling exclusions, cost, and an example. It is slightly longer than strictly necessary because the example overlaps somewhat with the schema, but every sentence earns its place and none merely restate the name or schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 10-parameter tool with an output schema and rich annotations, the description covers the essential context: what the tool returns conceptually, when to use alternatives, the cost, and an example. It does not explain the domain_filter or search parameters, but those are fully documented in the schema, so the description is complete enough for routing and correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all 10 parameters. The description adds minimal parameter semantics beyond the schema, mostly via the example invocation that references include_sitemap and max_urls. This matches the baseline of 3 when the schema carries the parameter documentation burden.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'list a site's URLs without fetching page bodies.' It explicitly distinguishes itself from siblings by naming what it is not for ('not for page content (scrape, or crawl_deep for many pages) and not for the links on one page (extract_links)'). An agent can immediately tell this apart from scrape, crawl_deep, and extract_links.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives explicit when-to-use guidance ('list a site's URLs without fetching page bodies') and explicit exclusions with named alternatives. It also includes cost and a concrete example, which helps an agent decide if invoking this tool is appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
process_documentARead-onlyIdempotent
Use this to extract text from a PDF or DOCX URL or file - research papers, contracts, reports. The body decides how it is read: a PDF or Word document served under sourceType "url" still reaches its parser, and a body this tool cannot read (an image, an archive) is refused by name. Returns structured sections, metadata, and word count. Not for ordinary web pages (scrape), though an HTML URL is accepted. Cost: 2 credits. Example: process_document({source: "https://example.com/report.pdf", sourceType: "pdf_url"})
| Name | Required | Description | Default |
|---|---|---|---|
| source | Yes | Document source - URL or file path | |
| options | No | Additional processing options (maxPages, pageRange:{start,end}, extractText, extractMetadata, outputFormat, ...) | |
| redact_pii | No | Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off | |
| sourceType | No | Type of document source | |
| user_agent | No | Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with. | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. | |
| max_inline_chars | No | Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and idempotentHint=true, and the description does not contradict them. Beyond that, it discloses cost (2 credits), the refusal of unsupported body types ('refused by name'), the fact that a PDF under sourceType 'url' still reaches its parser, and the return contents (structured sections, metadata, word count). This is substantial additional behavioral context that aids the agent in understanding side effects and edge cases.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the primary use case in the first sentence, then adds supporting details (edge cases, alternative, cost, example). It is slightly verbose but every sentence carries unique information. The structure is logical and it remains readable without being padded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's moderate complexity (7 params, nested object, enum, cost, redaction behavior), the description covers the key decision points: what it does, what it doesn't accept, an example, and the output shape. The output schema is not provided, but the description mentions return fields. It lacks explicit details on all option combinations, but the schema descriptions fill that gap. It is complete enough for an agent to call it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% for all 7 parameters, so the structured descriptions are comprehensive. The description adds value by providing an illustrative example (source + sourceType), clarifying the nuance that sourceType 'url' can still process PDFs, and implying the meaning of source and sourceType through the example. While it doesn't detail options like maxPages or redact_pii (handled in schema), this is above the baseline for high-coverage schemas.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('extract text') and resource ('PDF or DOCX URL or file'), names the target document types, and gives an example call. It explicitly distinguishes itself from web scraping ('Not for ordinary web pages (scrape)') and clarifies an edge case (HTML URL accepted but handled differently). This is precise and actionable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use this tool ('research papers, contracts, reports') and explicitly excludes ordinary web pages ('Not for ordinary web pages (scrape)'). It also gives a concrete usage example. It does not mention alternative tools by name (e.g., extract_text) but the exclusion and example are sufficient for an agent to select it correctly in most cases.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
read_resultARead-onlyIdempotent
Use this to read a result that came back with truncated: true and a result_handle - the tool kept the whole result for 1 hour and returned a preview. operation:"search" finds a literal query with offsets and context, "slice" returns characters from an offset, "lines" pages by line, "json_path" reads one subtree of a JSON result (crawl_deep pages, batch results, a fetch_url JSON body). Not a fetching tool: never call the original tool again while the handle is valid, and not for a result that arrived whole. Cost: 1 credit. Example: read_result({handle: "res_…", operation: "search", query: "pricing"})
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | json_path: dotted keys and array indexes, e.g. "results[3].content" — not JSONPath | |
| query | No | search: the text to find, matched literally, case-insensitive | |
| handle | Yes | The result_handle a truncated result returned (res_… or a batch id) | |
| length | No | slice: characters to return (default 10,000); lines: lines to return (default 200, max 5,000) | |
| offset | No | slice: first character (default 0); lines: first line index (default 0) | |
| operation | Yes | slice: characters from offset; search: case-insensitive literal query with context and offsets; lines: a page of lines; json_path: one subtree of a JSON result | |
| max_matches | No | search: matches to return (default 20) | |
| max_inline_chars | No | Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS) |
Output Schema
| Name | Required | Description |
|---|---|---|
| path | No | |
| text | No | slice: verbatim view.slice(offset, offset + length) |
| tool | No | The tool that produced the stored result |
| view | No | |
| _cost | No | Cost-transparency metadata (D3.5), present when injected into the text copy of the result |
| lines | No | |
| query | No | |
| value | No | json_path: the subtree; null with a preview when it is over max_inline_chars |
| handle | No | |
| length | No | slice: characters returned |
| offset | No | slice: first character returned |
| matches | No | search: matches with 200 chars of context each side |
| preview | No | |
| has_more | No | slice/lines: more follows the returned range |
| warnings | No | |
| operation | No | |
| truncated | No | search: more matches than returned; json_path: value replaced by a preview |
| view_path | No | |
| expires_at | No | |
| first_line | No | |
| line_count | No | |
| char_offset | No | lines: view offset of the first returned line |
| total_chars | No | Length of the full view |
| total_lines | No | |
| value_chars | No | |
| total_matches | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, so the safety profile is covered. The description adds valuable behavioral context: the 1-hour retention window, the preview behavior, the credit cost, and the semantics of each operation (search, slice, lines, json_path). This goes beyond the annotations without contradicting them.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single dense paragraph that front-loads the primary use case and ends with an example. It is somewhat lengthy but every sentence earns its place by covering scope, operations, exclusions, cost, and an example. A more structured layout could improve skimmability, but it remains efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given an output schema exists (not shown but present), the description need not detail return formats. It covers the main trigger, all operations, exclusions, cost, and handle validity. It does not explicitly mention error cases (e.g., expired handle), but the output schema likely handles those. Overall, it's sufficiently complete for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so every parameter is documented. The description adds operation-specific meaning (e.g., 'search finds a literal query with offsets and context', 'json_path reads one subtree') and a usage example that clarifies how parameters combine. This enriches understanding beyond the schema's basic field descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool reads truncated results, identifies the trigger condition (truncated: true with a result_handle), and explicitly distinguishes itself from fetching tools by instructing not to call the original tool again. It also names sibling tools indirectly and lists distinct operations, making its scope unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit when-to-use (truncated results) and when-not-to-use (whole results, not a fetching tool) guidance. It also gives a concrete example call and notes the 1-hour handle validity, which helps the agent decide when to invoke this tool versus alternatives like fetch_url or crawl_deep.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
reddit_searchARead-onlyIdempotent
Use this to search Reddit posts or comments, or read a full comment thread - reddit.com blocks direct scraping, so this reads the Arctic Shift community archive instead (free, no Reddit credentials). Modes: 'posts' (default) and 'comments' search; 'thread' returns a post plus its nested comment tree by link_id. A subreddit/author-scoped search queries the archive directly. A keyword search across ALL of Reddit finds posts with a site-restricted web search and then reads those posts from the archive, because Arctic Shift can only keyword-search within a scope; results come back as real archive rows, ordered by search relevance. An unscoped COMMENT search discovers posts the same way and then searches each post's comments for the keywords. A scoped comment search Arctic Shift times out on is retried over narrower windows (7d, 3d, 1d) and reports window_applied. Arctic Shift is tried first and the PullPush archive second for posts/comments searches (fallback_used says so; PullPush has refused automated clients since August 2026). Not for reddit.com URLs via scrape or fetch_url (blocked) - use mode:'thread' with the post's link_id. Cost: 5 credits. Example: reddit_search({query: "best mechanical keyboard", subreddit: "MechanicalKeyboards", limit: 10})
| Name | Required | Description | Default |
|---|---|---|---|
| mode | No | What to search: posts (default), comments, or thread (full comment tree — requires link_id) | |
| sort | No | Sort by post date (default desc = newest first) | |
| after | No | Only content posted after this date — ISO 8601, epoch seconds, or an offset like '7d' | |
| limit | No | Max results (default 25; thread mode: max comments returned) | |
| query | No | Keyword search. Posts: matches title+selftext; comments: matches body. Supports "quoted phrases", OR, -exclusion | |
| author | No | Limit to one author (with or without the u/ prefix) | |
| before | No | Only content posted before this date — same formats as after | |
| source | No | Backend: auto routes + falls back (default). web_discovery serves only unscoped keyword searches (web search finds the posts, the archive supplies the rows). reddit_api uses the official Reddit Data API — only when REDDIT_CLIENT_ID/REDDIT_CLIENT_SECRET are set; serves posts/thread, not comment search | |
| link_id | No | Post ID (e.g. '1twm1zh' or 't3_1twm1zh') — required for thread mode, optional filter for comments mode | |
| subreddit | No | Limit to one subreddit (with or without the r/ prefix) |
Output Schema
| Name | Required | Description |
|---|---|---|
| mode | No | |
| post | No | thread mode: the post itself |
| _cost | No | Cost-transparency metadata (D3.5), present when injected into the text copy of the result |
| count | No | |
| notes | No | Data-provenance caveats (archive freshness, coverage gaps) |
| query | No | |
| author | No | |
| source | No | Which backend served this result — an archive, or web discovery (site-restricted web search hydrated from the archive) for Reddit-wide keyword search |
| link_id | No | Present in thread mode |
| results | No | posts/comments modes |
| comments | No | thread mode: nested comment tree ({...comment, replies:[...]}); collapsed branches appear as {more_count, more_ids} |
| checkedAt | No | |
| subreddit | No | |
| discovered | No | web_discovery: how many post ids the site-restricted web search surfaced before archive hydration |
| comment_count | No | |
| fallback_used | No | Present when the primary archive failed and the fallback served the result |
| posts_searched | No | web_discovery comments mode: how many discovered posts had their comments searched before limit was reached |
| window_applied | No | arctic_shift comments mode: the after-window ("7d"/"3d"/"1d") the search was narrowed to after the full-history search timed out; absent when the caller set after or no narrowing was needed |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations (readOnlyHint, idempotentHint, etc.), the description discloses substantial behavioral traits: it reads from Arctic Shift and PullPush archives, explains that Reddit blocks direct scraping, details fallback logic with fallback_used, describes timeout retry windows (7d, 3d, 1d) and window_applied, mentions PullPush's refusal of automated clients since August 2026, and notes the cost of 5 credits. This adds meaningful context that annotations alone cannot convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but information-dense and front-loaded with the core purpose and mode overview. It covers many nuances without redundancy. It could be slightly trimmed, but every sentence earns its place given the tool's complexity, and the structure flows logically from general usage to specifics like fallbacks and exclusions.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is comprehensive for a tool with 10 parameters and 3 enums. It explains all modes, backend behaviors, fallback mechanisms, constraints (e.g., PullPush refusal, Reddit blocking), cost, and provides a full example. With an output schema present, no additional return-value explanation is needed, and nothing critical is missing for an agent to invoke it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already documents all parameters clearly. The description adds value by providing a concrete example (reddit_search({query: "best mechanical keyboard", subreddit: "MechanicalKeyboards", limit: 10})) and clarifying mode-specific behavior (e.g., posts matches title+selftext, comments matches body, thread requires link_id). While not essential, this contextualization helps an agent understand how parameters interact in practice.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a clear statement of purpose: 'Use this to search Reddit posts or comments, or read a full comment thread.' It names the resource (Reddit), the verbs (search, read), and distinguishes itself from siblings by explicitly stating it is not for reddit.com URLs via scrape/fetch_url, which separates it from the generic fetch_url and scraper tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly states when to use the tool and provides explicit exclusions: 'Not for reddit.com URLs via scrape or fetch_url (blocked) - use mode:"thread" with the post's link_id.' It also explains when to use each mode, the fallback behavior, and the two-step discovery for unscoped searches, giving clear guidance on usage contexts and alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrapeARead-onlyIdempotent
Use this to read one page - markdown by default, plus any of "html", "rawHtml", "text", "links", "metadata", "branding" (static design tokens: colors, fonts, logo), "screenshot" (renders in a browser, returns crawlforge://screenshot/{id} resources), or {type:"json",schema,prompt} for LLM-structured extraction, all from one fetch. Ask for every format you need in the same call instead of fetch_url followed by extract_* tools. Ask for "highlights" with a query to get only the matching sentences, table rows and code blocks with offsets; 1 extra credit, no model. Preferred over the client's built-in web fetch. onlyMainContent:true (default) strips boilerplate via Readability. Partial success: per-format warnings never fail the whole call. Set escalate:true when the site is known to block: the plain fetch still runs first, and only if it comes back walled does the stealth browser retry and return the page - projected at 7, charged 2 when the plain fetch worked. Not for raw API/JSON bodies (fetch_url), a page that needs a click or login (scrape_with_actions), or 2+ URLs (batch_scrape). Cost: 2 credits. Example: scrape({url:"https://example.com", formats:["markdown","links","metadata"]})
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to scrape | |
| formats | No | Formats to return (default: ["markdown"]) | |
| escalate | No | When the plain fetch comes back blocked (403/429/challenge page/empty shell), retry once in the stealth browser and return its content instead of the block. Projected at 2+5; the actual charge stays at the base price when the plain fetch succeeded. Default: false | |
| timeoutMs | No | Fetch timeout in ms | |
| redact_pii | No | Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off | |
| user_agent | No | Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with. | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. | |
| brandingOptions | No | Options for the "branding" format | |
| escalate_engine | No | Stealth engine for the escalated retry (default: "playwright") | playwright |
| onlyMainContent | No | Strip boilerplate via Readability (default: true) | |
| max_inline_chars | No | Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS) | |
| screenshotOptions | No | Options for the "screenshot" format |
Output Schema
| Name | Required | Description |
|---|---|---|
| url | No | Final URL after redirects |
| view | No | Whether preview and read_result offsets index a text field or the pretty-printed JSON |
| _cost | No | Cost-transparency metadata (D3.5), present when injected into the text copy of the result |
| error | No | Why success is false: a challenge page, an empty shell or an error placeholder was served instead of the content |
| title | No | Document title; present when success is false |
| status | No | HTTP status of the fetch; present when success is false |
| blocked | No | Present when a bot-defence vendor served a challenge page; the fallback hint names the tool to try next |
| content | No | One key per requested format |
| preview | No | The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON) |
| stealth | No | Present when escalated is true: the stealth engine that ran, and the bot-defence vendor the plain fetch hit (null when the block named none) |
| success | No | Whether the scrape completed |
| warnings | No | Per-format warnings; partial success never fails the whole call |
| escalated | No | Present only when escalate:true was passed: whether the blocked plain fetch was retried in the stealth browser. False means the plain fetch sufficed and the call is charged at the base price |
| redaction | No | Present when redact_pii was set: what was redacted from the text of this result |
| truncated | No | True when the inline result is a preview |
| view_path | No | Dotted path of the text field the view was cut from; null for the JSON view |
| expires_at | No | When the stored result is dropped (ISO 8601) |
| total_chars | No | Length of the full view in characters |
| result_handle | No | Handle for read_result; the full result is kept 1 hour |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare readOnlyHint=true, idempotentHint=true, destructiveHint=false; the description adds rich behavior beyond them: per-format warnings never fail the whole call, onlyMainContent strips via Readability, escalate runs the plain fetch first and only retries in the stealth browser when walled, plus cost and example. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Long but dense: every sentence earns its place. Purpose and core behavior are front-loaded, then formats, exclusions, cost, and an example. The grouped clause style keeps a high-information paragraph readable without waste.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 12-parameter tool with nested objects and multiple formats, the description covers selection criteria, exclusions, cost model, partial-success behavior, escalation, and an example. An output schema exists, so return values need no explanation; the schema fills any remaining parameter-level details.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description adds real meaning beyond the schema: explains branding as static design tokens, screenshot returns crawlforge:// resources, highlights query costs 1 extra credit with no model, and gives a concrete call example. It does not cover every parameter, but the schema already documents them well.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Opens with a specific verb and resource: 'Use this to read one page'. Enumerates the exact formats, and explicitly distinguishes from siblings: 'Not for raw API/JSON bodies (fetch_url), a page that needs a click or login (scrape_with_actions), or 2+ URLs (batch_scrape)'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
States when to use this tool and names concrete alternatives with the conditions that select them: fetch_url for API/JSON, scrape_with_actions for click/login, batch_scrape for 2+ URLs. Also says 'Preferred over the client's built-in web fetch' and tells when to set escalate:true.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrape_structuredARead-onlyIdempotent
Use this when you know the exact CSS selectors for the data you want - e.g. a pricing table or product list with consistent markup. More reliable than LLM extraction for well-structured pages. By default each selector is matched independently across the whole page, so the returned arrays are NOT row-aligned: data.price[0] need not belong to the same row as data.name[0]. Pass row_selector to get aligned records instead - one object per row, null for a field the row lacks. Not for pages whose markup varies or where you cannot name the selectors (extract_structured, LLM-driven). Cost: 2 credits. Example: scrape_structured({url: "https://shop.com/products", row_selector: ".product-card", selectors: {price: ".price", name: ".product-title"}})
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to scrape | |
| selectors | Yes | CSS selectors mapping field names to selectors. Append @attr to extract an attribute instead of text (e.g. "a.link@href", "img@src") | |
| user_agent | No | Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with. | |
| max_results | No | Maximum number of matches to return per field when a selector matches multiple elements, or the maximum number of rows when row_selector is set | |
| row_selector | No | CSS selector for the repeating row/container element. When set, each field in selectors is matched inside each row and data is an array of row-aligned records ({field: value|null}) instead of parallel arrays | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnly/idempotent annotations, it discloses the critical row-alignment gotcha (data.price[0] need not belong to the same row as data.name[0]), the cost of 2 credits, and the meaning of row_selector. This is exactly the kind of non-obvious behavior an agent needs before invoking.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description leads with the decision rule, then covers exclusions, the key behavioral warning, cost, and an example with no filler. Every sentence contributes necessary operational information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite having no output schema, the description explains the shape of returned data (parallel arrays vs row-aligned objects) and gives enough context for an agent to invoke correctly. Together with the rich input schema, this is complete for a read-only extraction tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already covers all 6 parameters in detail, so the baseline is 3. The description adds value with a concrete usage example and clarifies how selectors and row_selector work together, but it does not substantially redefine the parameter meanings beyond the schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb and resource (scrape structured data with exact CSS selectors) and differentiates it from LLM-driven extraction. It is immediately clear this tool is for well-structured pages with known markup.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives an explicit when-to-use condition (you know exact selectors, consistent markup) and states what it is not for (varying markup or unnamed selectors), pointing to extract_structured / LLM-driven extraction as the alternative. This lets an agent select correctly among many siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrape_templateARead-onlyIdempotent
Use this when you want structured data from a well-known site or platform API without writing custom selectors. Three modes: a template id with a url (scrape_template({template:"github-repo", url:"https://github.com/user/repo"})); template:"auto" with a url, which picks the template from the URL and names its choice in the response; or template:"list" to enumerate every template with the URLs it handles. Page templates return one record - e-commerce, social, developer and news sites (shopify-product, amazon-product, github-repo, youtube-video, reddit-thread, hacker-news-front-page, producthunt-launch, stackoverflow-question, npm-package; reddit-thread reads the post from the Arctic Shift archive and reddit_search reads the comment tree). linkedin-profile and tweet are retired - those sites' robots.txt disallow every keyless path - and naming one returns the reason. List connectors return N records from one call and are driven by params instead of a url: job boards (Greenhouse, Lever, Ashby, Workable, Recruitee, Teamtailor) return a company's whole careers board, US government APIs (NHTSA VIN decode, NPI provider registry) answer keyless lookups, and shopify-collection returns a whole collection. Not for a site without a template (scrape) - template:"list" shows what exists. Cost: 1 credit. Example: scrape_template({template:"greenhouse-jobs", params:{company:"stripe"}})
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | URL to scrape — required unless template is list, or params drive a list connector | |
| params | No | Parameters for a list connector, e.g. {company:"stripe"} for greenhouse-jobs or {store:"www.allbirds.com", collection:"mens"} for shopify-collection. Use template:"list" to see which templates take params | |
| timeout | No | Request timeout in milliseconds | |
| template | Yes | Template ID (e.g. github-repo), "auto" to detect one from the url, or "list" to enumerate available templates | |
| user_agent | No | Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with. | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond annotations (readOnlyHint, openWorldHint, idempotentHint), the description discloses cost (1 credit), robots.txt handling including the recording of respect_robots:false against the API key, the retired-template behavior with the reason returned, auto-mode naming its template choice, and record-count differences between page and list templates. No contradiction with annotations exists.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but front-loaded with the purpose and organized into modes, template catalogs, retired templates, list connectors, exclusions, cost, and example. Every sentence earns its place in terms of content, but the density is high enough that a few aggressive editors might request trimming; it remains well above the minimum viable level.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema present, the description compensates by stating return shapes: 'one record' for page templates, 'N records' for list connectors, 'names its choice' in auto mode, and 'returns the reason' for retired templates. It also covers cost, robots behavior, a full parameter-driven example, and the full catalog of supported sites, making it complete for an agent to call correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Although schema coverage is 100%, the description adds substantial meaning: it defines the three template modes, clarifies when url is required, gives concrete template IDs for both page and list connectors, and provides a full example invocation with params. It also explains philosophy of user_agent honesty and respect_robots decisions, which the schema alone does not convey.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'Use this when you want structured data from a well-known site or platform API without writing custom selectors,' naming the verb, resource, and value proposition. It differentiates itself from siblings by explicitly naming scrape as the alternative for sites without a template, and by enumerating exactly which templates exist. No ambiguity remains about what the tool produces.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It starts with 'Use this when...' and later states 'Not for a site without a template (scrape),' giving an explicit exclusion and naming the alternative tool. It further explains when to use template:'list' to discover what exists and how params-driven list connectors differ from url-driven page templates, leaving no strategic decision to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrape_with_actionsARead-only
Use this when you must interact with a page before scraping - login, click buttons, fill forms, scroll, or wait for dynamic content to load - for SPAs, login-gated content, or multi-step flows. Actions: snapshot, wait, click, type, press, scroll, screenshot, executeJavaScript, select (dropdowns), hover, navigate. Start a chain with {type:"snapshot"} to list the page's interactive elements with stable refs (@e1, @e2 ...), then target those refs in later actions instead of guessing CSS selectors; navigation invalidates refs, so snapshot again after one. Set browserOptions.stealth:true to run the chain in the stealth browser. robots.txt is respected on every navigation. Screenshots from this tool are stored as crawlforge://screenshot/{actionId} resources. Not for pages that render without interaction (scrape) and not as the first attempt on a blocked site (stealth_mode operation:"scrape"). Cost: 5 credits. Example: scrape_with_actions({url: "https://app.com/dashboard", actions: [{type:"snapshot"},{type:"type",selector:"@e2",text:"user@a.com"},{type:"click",selector:"@e4"}]})
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to scrape | |
| actions | Yes | Browser actions to perform before scraping | |
| formats | No | Output formats for scraped content | |
| maxRetries | No | Maximum retry attempts on failure | |
| redact_pii | No | Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off | |
| formAutoFill | No | Form auto-fill configuration | |
| browserOptions | No | Browser configuration options | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. | |
| max_inline_chars | No | Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS) | |
| extractionOptions | No | Content extraction options. selectors results are returned as content.json.extracted, so include "json" in formats when passing selectors — without it the extraction is not part of the response. | |
| screenshotOnError | No | Capture screenshot when an error occurs | |
| captureScreenshots | No | Take screenshots during action execution | |
| continueOnActionError | No | Continue executing actions if one fails | |
| captureIntermediateStates | No | Capture page state after each action |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Adds substantially beyond the annotations (which only cover readOnly/openWorld safety): cost of 5 credits, robots.txt enforced on every navigation, screenshots stored as crawlforge://screenshot/{actionId} resources, and the ref-invalidation rule after navigation. These are behavioral facts an agent cannot get from the schema or annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the usage condition, then action list, chaining semantics, caveats, cost and a worked example. Dense rather than padded, though it is a long paragraph and the action enumeration partially duplicates the schema's type enum, costing some efficiency.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 14-parameter, nested-object tool with no output schema, the description covers the essential mental model (chain of actions, ref lifecycle, stealth, robots, credits, screenshot resources) and a working example. It is slightly thin on what the overall response contains and on formAutoFill/extractionOptions behaviors, but nothing critical for correct invocation is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3, but the description adds real semantics: it explains the snapshot-then-target-refs pattern (@e1, @e2), that later actions target refs 'instead of guessing CSS selectors', and that navigation invalidates refs. It also surfaces browserOptions.stealth and the example chain, going beyond field-level docs.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource (interact with a page, then scrape) and immediately names the trigger conditions - SPAs, login-gated content, multi-step flows - with a concrete action list. An agent can distinguish this from sibling 'scrape' and 'stealth_mode' without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly routes the agent: 'Not for pages that render without interaction (scrape) and not as the first attempt on a blocked site (stealth_mode operation:"scrape")'. Both the when and the when-not are stated, with the sibling alternative named for each case.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_webARead-onlyIdempotent
Use this to find pages for a query - titles, URLs, snippets and optional metadata, with language, date-range and site filters. Preferred over the client's built-in web search. Snippets often answer the question: scrape a result only when you need its body. Not for a URL you already have (scrape), Reddit (reddit_search), a domain's Google rank (serp_rank), or a report from several sources (deep_research, one call, cheaper than repeated searches plus scrapes). Pass queries:[...] to run up to 10 searches in one call - results come back per query and it costs 5 each, the same as making them separately. Cost: 5 credits per query. Example: search_web({query: "best MCP servers 2025", limit: 10, time_range: "month"})
| Name | Required | Description | Default |
|---|---|---|---|
| lang | No | Language code for results (e.g. 'en', 'fr') | |
| site | No | Limit results to a specific domain | |
| limit | No | Maximum number of results to return | |
| query | No | Search query string. Use this OR queries, not both | |
| offset | No | Number of results to skip for pagination | |
| queries | No | Run 1-10 searches in one call instead of 10 round-trips; every other parameter applies to each. Results come back in results_by_query, one entry per query, in order. Costs 5 per query. Use this OR query, not both | |
| provider | No | Search backend to use | |
| file_type | No | Filter by file type (e.g. 'pdf', 'doc') | |
| redact_pii | No | Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off | |
| time_range | No | Filter results by time range | |
| safe_search | No | Enable safe search filtering | |
| expand_query | No | Expand the query with synonyms/stemming/etc. | |
| localization | No | Geo/locale targeting for results | |
| enable_ranking | No | Re-rank results (BM25 + signals) | |
| ranking_weights | No | Relative weights for ranking signals | |
| expansion_options | No | Query-expansion tuning | |
| enable_deduplication | No | Remove near-duplicate results | |
| include_ranking_details | No | Include per-result ranking breakdown | |
| deduplication_thresholds | No | Similarity thresholds for dedup | |
| include_deduplication_details | No | Include dedup decision details |
Output Schema
| Name | Required | Description |
|---|---|---|
| _cost | No | Cost-transparency metadata (D3.5), present when injected into the text copy of the result |
| count | No | Batch form: how many queries ran |
| limit | No | |
| query | No | |
| cached | No | |
| offset | No | |
| queries | No | Batch form: the queries that ran, in order |
| results | No | |
| provider | No | |
| redaction | No | Present when redact_pii was set: what was redacted from the text of this result |
| processing | No | |
| search_time | No | |
| localization | No | |
| total_results | No | |
| effective_query | No | Present when query expansion changed the query actually used |
| expanded_queries | No | |
| results_by_query | No | Batch form: one entry per query, in order |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already cover read-only, idempotent, non-destructive behavior. The description adds meaningful behavior beyond that: cost of 5 credits per query, batched queries returning per-query results, and the useful behavioral hint that snippets often answer the question directly. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Every sentence earns its place: purpose, filter options, preference over built-in search, alternatives, batching behavior, cost, and an example. The most important purpose statement is front-loaded, and the description is dense without being padded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the rich schema (100% parameter coverage, output schema present) and strong annotations, the description fills the remaining gaps effectively: when to use which sibling, batching semantics, cost, and a usage example. An agent has everything needed to invoke this tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the baseline is 3 even without additional description. The description reinforces parameters like query, limit, time_range, and queries, and gives a concrete example, but it mostly restates what the schema already documents rather than adding new semantic depth.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'find pages for a query' and explicitly lists the return contents (titles, URLs, snippets, optional metadata). It clearly distinguishes itself from siblings by naming search_web, scrape, reddit_search, serp_rank, and deep_research as different tools for different tasks.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit when-to-use ('Preferred over the client's built-in web search'), when-not-to-use ('Not for a URL you already have', Reddit, Google rank, or multi-source reports), and names the exact alternative tool for each case. Also advises scraping only when the snippet is insufficient, which is practical routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
serp_rankARead-onlyIdempotent
Use this to check where a domain ranks in Google's ORGANIC results for a keyword - real SERP position, not Custom Search order. Returns the target's organic rank, the ranking URL, and every position it holds. Not for general search (search_web). Requires DataForSEO credentials and returns configured:false without them - do not retry in that case. Cost: 5 credits (0 when unconfigured). Example: serp_rank({keyword: "managed wordpress hosting", target: "dashboardhosting.com", location_name: "United States"})
| Name | Required | Description | Default |
|---|---|---|---|
| depth | No | How many results to scan, 10-200 (default 20; DataForSEO bills ~$0.002 per 10 and gets slower the deeper it goes) | |
| device | No | Device to emulate | |
| target | Yes | Domain or URL to locate in the results (e.g. 'example.com') | |
| keyword | Yes | The search query to check ranking for | |
| language_code | No | Language code (e.g. 'en') | |
| location_code | No | Numeric DataForSEO location code (overrides location_name) | |
| location_name | No | Location, e.g. 'United States' or 'London,England,United Kingdom' |
Output Schema
| Name | Required | Description |
|---|---|---|
| url | No | URL of the target's best-ranking result |
| cost | No | USD charged by DataForSEO for this lookup (separate from CrawlForge credits) |
| note | No | Present when configured=false, explains how to enable |
| _cost | No | Cost-transparency metadata (D3.5), present when injected into the text copy of the result |
| found | No | Whether the target appeared anywhere in the scanned SERP |
| title | No | |
| device | No | |
| target | No | Bare target domain, normalized |
| keyword | No | |
| results | No | Top organic competitors as Google actually ranks them (capped) |
| checkUrl | No | Link to view the real SERP on DataForSEO |
| location | No | |
| position | No | Best (lowest) organic rank; null = not within top `depth` |
| checkedAt | No | |
| configured | No | False when DATAFORSEO_LOGIN/PASSWORD are unset — no rank was fabricated |
| allPositions | No | Every position the target holds on this SERP |
| depthScanned | No | |
| rankAbsolute | No | |
| organicResults | No | |
| seResultsCount | No |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already establish readOnly/idempotent/non-destructive safety, so the bar is lower; the description adds valuable extras: external DataForSEO dependency, credential requirement, unconfigured failure mode, cost implications (5 credits, 0 unconfigured), and a no-retry rule. It does not disclose rate limits or pagination, but those are not necessary for a read-only lookup with an output schema.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is tight and front-loaded: the core purpose appears first, followed by the sibling exclusion, operational caveat, cost, and example. Every sentence earns its place and none repeats the schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a read-only external-SERP lookup with full schema coverage and an output schema, the description covers what an agent needs: purpose, scope, credentials, failure handling, cost, and example inputs. No critical behavioral or invocation context is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all 7 parameters. The description mainly reinforces keyword/target semantics and gives a concrete example with location_name, but does not add much meaning beyond that. Baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and object: 'check where a domain ranks in Google's ORGANIC results for a keyword.' It explicitly distinguishes from Custom Search order and names the sibling search_web as the wrong tool for general search, so an agent can select it accurately.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives explicit routing guidance: 'Not for general search (search_web).' It also supplies operational prerequisites (DataForSEO credentials), defines the unconfigured failure value (configured:false), and tells the agent not to retry in that case. An example invocation further clarifies expected usage.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
stealth_modeA
Use this when a site blocks normal scraping - Cloudflare, Datadome, or other bot-detection systems. Renders in a Playwright browser with randomized fingerprints, human behavior simulation, WebRTC/canvas spoofing. operation:"scrape" is the one-shot path: it creates a context, navigates, returns the requested formats and tears down. The create_context -> create_page -> cleanup operations remain for multi-step work. robots.txt is respected on every navigation. Not a first choice: try scrape first and switch here after a 403/429/CAPTCHA/challenge page or an empty shell. Cost: 5 credits per browser operation; configure, enable, disable, get_stats and cleanup cost 1. Example: stealth_mode({operation:"scrape", url:"https://example.com", formats:["markdown","links"]})
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | URL to scrape — required for operation:"scrape" | |
| engine | No | Browser engine: "playwright" (Chromium, default) or "camoufox" (Firefox-based, higher anti-detect score — install with npm install camoufox) | playwright |
| formats | No | Formats to return from operation:"scrape" (default: ["markdown"]). "screenshot" returns a crawlforge://screenshot/{id} resource URI. | |
| verbose | No | Return the full generated fingerprint from create_context instead of a summary | |
| wait_for | No | Extra wait after page load, in ms — for content that renders after DOMContentLoaded | |
| contextId | No | Browser context ID for page operations | |
| operation | No | Stealth operation to perform | configure |
| urlToTest | No | URL to navigate to when creating a page | |
| redact_pii | No | Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off | |
| stealthConfig | No | Stealth browser configuration with anti-detection settings | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. | |
| max_inline_chars | No | Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond annotations that only flag readOnly=false, the description discloses concrete behavioral traits: randomized fingerprints, human behavior simulation, WebRTC/canvas spoofing, robots.txt respect on every navigation, one-shot teardown, and credit costs per operation. It also clarifies the side-effect of setting respect_robots to false (recorded against the API key). These add value beyond what annotations convey, though it doesn't cover rate limits or response error shapes.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is one dense paragraph, but every sentence carries decision-relevant information: when to use, what it does, operation modes, robots handling, cost model, and an example. Some structure (bullets for cost or operation modes) would improve scannability, but there is no filler or repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 12 parameters, nested objects, multiple operations, and no output schema, the description covers the key contextual elements: the anti-bot problem it solves, the two usage patterns, robots.txt policy, credit cost, and an example. It does not explicitly state what the return payload looks like beyond 'returns the requested formats,' but the schema already describes max_inline_chars, redact_pii, and result_handle, so the missing piece is minor.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so parameters are already documented, but the description enriches the operation parameter by explaining the one-shot scrape lifecycle versus the multi-step alternatives and gives a concrete invocation example ('stealth_mode({operation:"scrape", url:"https://example.com", formats:["markdown","links"]})'). It also attaches cost semantics to specific operations (5 credits per browser operation vs 1 for configure/enable/disable/get_stats/cleanup), meaning beyond the raw enum values in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific trigger ('when a site blocks normal scraping – Cloudflare, Datadome, or other bot-detection systems') and names the resource and action ('Renders in a Playwright browser... operation:"scrape"'). It also explicitly distinguishes itself from the normal 'scrape' sibling with 'Not a first choice: try scrape first and switch here after a 403/429/CAPTCHA/challenge page or an empty shell.' This gives an agent a precise selection basis.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides explicit when-to-use guidance ('Use this when a site blocks normal scraping...'), a clear alternative ('try scrape first'), and the exact failure conditions that justify switching (403/429/CAPTCHA/challenge page or empty shell). It also separates the one-shot scrape path from the multi-step create_context -> create_page -> cleanup sequence, so an agent knows which operations fit single vs repeated work.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
summarize_contentARead-onlyIdempotent
Use this to condense text you already hold into a briefing, comparison, or shorter LLM context - extractive (sentence selection) or abstractive (rewrite via Ollama/sampling). Takes text, not a URL: pass the markdown from a scrape result. Not needed for text short enough to summarise in context yourself. Cost: 4 credits. Example: summarize_content({text: "..long article..", options: {summaryLength: "short", summaryType: "abstractive"}})
| Name | Required | Description | Default |
|---|---|---|---|
| text | Yes | The text content to summarize | |
| options | No | Summarization options |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare the safety profile (readOnlyHint, idempotentHint, destructiveHint false). The description adds meaningful behavioral context: it accepts text not URLs, performs extractive or abstractive summarization, leverages Ollama/sampling for abstractive rewrites, and costs 4 credits. This goes beyond what annotations provide, though it doesn't disclose return format or edge-case behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is dense but organized: it starts with the core purpose, then input constraint, a usage heuristic, cost, and a concrete example. No sentence is wasted; the structure front-loads the main action and defers cost and example details.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter tool with high schema coverage but an empty options schema, the description covers how to invoke it (text plus optional options), what input type to pass, and an example. It lacks an explicit statement of return value shape, which matters because there is no output schema; however, the core invocation requirements are adequately specified.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% for both properties, but the 'options' property is an empty object in the schema, providing no usable semantics. The description compensates with an example showing summaryLength and summaryType, and clarifies that 'text' should be markdown from a scrape result. This adds real meaning beyond the schema, especially where the schema is empty.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('condense') and names the resource ('text you already hold'), and distinguishes itself from URL-input tools by stating 'Takes text, not a URL.' It also lists output forms (briefing, comparison, shorter LLM context) and methods (extractive/abstractive), which clarifies exactly what it does.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit when-to-use context: summarizing text already held, and an explicit when-not-to-use: 'Not needed for text short enough to summarise in context yourself.' It also directs users to pass markdown from a scrape result, implying the preceding step. It doesn't name a specific sibling tool as an alternative, but it gives enough routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
track_changesA
Use this to monitor a URL for content changes over time - competitor pricing, regulation updates, product availability. Start with operation:"create_baseline", then periodically use operation:"compare" to diff; repeated compare calls on the same URL are expected. Supports webhooks and scheduled monitoring, and scheduledMonitorOptions.hosted:true runs the monitor on CrawlForge's servers with email and signed webhooks. Not for a one-off read (scrape). Cost: 3 credits. Example: track_changes({url: "https://example.com/pricing", operation: "create_baseline"})
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | The URL to track changes for (optional for list_scheduled_monitors) | |
| html | No | HTML content to compare against baseline | |
| content | No | Content to compare against baseline | |
| operation | No | Tracking operation to perform | compare |
| user_agent | No | Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with. | |
| queryOptions | No | Query options for history and stats retrieval | |
| exportOptions | No | Export options for change history data | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. | |
| storageOptions | No | Storage and history retention settings | |
| trackingOptions | No | Options for how changes are tracked and compared | |
| alertRuleOptions | No | Alert rule configuration for change notifications | |
| dashboardOptions | No | Dashboard display options | |
| monitoringOptions | No | Monitoring schedule and notification settings | |
| notificationOptions | No | Notification configuration for webhooks, Slack and email (email is sent by hosted monitors only) | |
| scheduledMonitorOptions | No | Scheduled monitoring: recurring compare + notify, optional plain-English goal |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only disclose readOnlyHint=false, openWorldHint=true, idempotentHint=false, destructiveHint=false. The description supplements this with useful behavioral nuances: cost (3 credits), hosted CrawlForge servers with signed webhooks, and the expected pattern of repeated compare calls on the same URL. This goes beyond the annotation surface but does not detail side effects like state storage of baselines, which is part of the tool's open-world effect.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
It is only two sentences and still covers purpose, workflow, alternatives, cost, hosting option, and an example. Everything is relevant and tightly packed without redundancy; it reads like a well-specified instruction rather than an exhaustive manual.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is very complex (15 parameters, nested objects, numerous sub-options). The description gives a good high-level flow but does not describe return values (no output schema present) nor elaborate on each of the many parameter groups (e.g., queryOptions, storageOptions, alertRuleOptions). Since there is no output schema, the description is expected to explain what the tool returns, which is missing here. It does provide enough to decide when to use it, but not enough for full understanding of its behavior across all operations.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema carries all parameter descriptions. However the description adds a meaningful semantic: a clear workflow of operations (create_baseline, then compare) and a concrete example with url and operation. That enriches param selection beyond what the schema enum alone provides, so the description gives something extra.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states the tool's specific verb (monitor) and resource (a URL), plus concrete purposes (competitor pricing, regulation updates, product availability) to distinguish from a one-off scrape. It explicitly says 'Not for a one-off read (scrape)', which differentiates it from sibling tools like scrape or fetch_url without needing to inspect them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives an explicit when-to-use pattern: 'Start with operation: create_baseline, then periodically use operation: compare' and adds a when-not ('Not for a one-off read (scrape)'). It also references the alternative `scrape` directly, qualifying as explicit alternative rejection and a clear usage flow.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
2 tool updates
v6.6.0- Added
browser_session - Changed
scrape_with_actions4 fields changed- added
Input schema / properties / actions / items / properties / interactiveOnlyAdded value: +{ + "description": "snapshot: only interactive elements (default true)", + "type": "boolean" +} - added
Input schema / properties / actions / items / properties / maxNodesAdded value: +{ + "description": "snapshot: cap on nodes listed (default 200, max 1000); the result says truncated when the cap stopped the walk", + "maximum": 1000, + "minimum": 1, + "type": "number" +} - added
Input schema / properties / actions / items / properties / selector / descriptionAdded value: +"A CSS selector, or a @e1 ref from an earlier snapshot action in this chain" - changed
Input schema / properties / actions / items / properties / type / enumPrevious value: -[ - "wait", - "click", - "type", - "press", - "scroll", - "screenshot", - "executeJavaScript", - "select", - "hover", - "navigate" -]New value: +[ + "snapshot", + "wait", + "click", + "type", + "press", + "scroll", + "screenshot", + "executeJavaScript", + "select", + "hover", + "navigate" +]
4 tool updates
v6.5.0- Changed
extract_structured1 field changed- changed
Output schema / properties / success / descriptionPrevious value: -"False when the extraction errored or a required field came back missing or empty"New value: +"False when the extraction errored, or a required field came back missing, empty, or in the wrong shape"
- Changed
generate_llms_txt1 field changed- changed
Input schema / properties / outputOptions / properties / contactEmail / patternPrevious value: -"^(?!\\.)(?!.*\\.\\.)([A-Za-z0-9_'+\\-\\.]*)[A-Za-z0-9_+-]@([A-Za-z0-9][A-Za-z0-9\\-]*\\.)+[A-Za-z]{2,}$"New value: +"^(?:[A-Za-z0-9_'+\\-]+\\.)*[A-Za-z0-9_'+\\-]*[A-Za-z0-9_+-]@(?:[A-Za-z0-9][A-Za-z0-9\\-]*\\.)+[A-Za-z]{2,}$"
- Changed
get_batch_results1 field changed- added
Input schema / properties / max_inline_charsAdded value: +{ + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" +}
- Changed
track_changes9 fields changed- added
Input schema / properties / monitoringOptions / defaultAdded value: +{} - changed
Input schema / properties / notificationOptions / descriptionPrevious value: -"Notification configuration for webhooks and Slack"New value: +"Notification configuration for webhooks, Slack and email (email is sent by hosted monitors only)" - added
Input schema / properties / notificationOptions / properties / emailAdded value: +{ + "properties": { + "enabled": { + "default": false, + "type": "boolean" + }, + "includeDetails": { + "default": true, + "type": "boolean" + }, + "recipients": { + "items": { + "format": "email", + "pattern": "^(?:[A-Za-z0-9_'+\\-]+\\.)*[A-Za-z0-9_'+\\-]*[A-Za-z0-9_+-]@(?:[A-Za-z0-9][A-Za-z0-9\\-]*\\.)+[A-Za-z]{2,}$", + "type": "string" + }, + "type": "array" + }, + "subject": { + "type": "string" + } + }, + "type": "object" +} - added
Input schema / properties / queryOptions / defaultAdded value: +{} - added
Input schema / properties / scheduledMonitorOptions / properties / hostedAdded value: +{ + "default": false, + "description": "Run the monitor on CrawlForge's servers: it fires from the hosted scheduler whether or not this process is alive and sends email and signed webhooks. Each check bills 3 credits per compared target from the account; blocked and errored targets are free. Default false = local, in-process.", + "type": "boolean" +} - added
Input schema / properties / scheduledMonitorOptions / properties / nameAdded value: +{ + "description": "Display name for a hosted monitor (default: the URL host)", + "maxLength": 80, + "minLength": 1, + "type": "string" +} - added
Input schema / properties / storageOptions / defaultAdded value: +{} - added
Input schema / properties / trackingOptions / defaultAdded value: +{} - added
Input schema / properties / trackingOptions / properties / excludeSelectors / defaultAdded value: +[ + "script", + "style", + "noscript", + ".advertisement", + ".ad", + "#comments" +]
29 tool updates
v6.0.0- Changed
agent2 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - added
Input schema / properties / schema / propertyNamesAdded value: +{ + "type": "string" +}
- Changed
analyze_content2 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - changed
Input schema / properties / options / additionalPropertiesPrevious value: -trueNew value: +{}
- Changed
batch_scrape8 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - added
Input schema / properties / extractionSchema / propertyNamesAdded value: +{ + "type": "string" +} - removed
Input schema / properties / jobOptions / additionalPropertiesRemoved value: -false - added
Input schema / properties / max_inline_charsAdded value: +{ + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" +} - added
Input schema / properties / redact_piiAdded value: +{ + "anyOf": [ + { + "type": "boolean" + }, + { + "properties": { + "entities": { + "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected", + "items": { + "type": "string" + }, + "type": "array" + }, + "mode": { + "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "replace_style": { + "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value", + "enum": [ + "tag", + "mask", + "remove" + ], + "type": "string" + } + }, + "type": "object" + } + ], + "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off" +} - changed
Input schema / properties / urls / items / anyOfPrevious value: -[ - { - "format": "uri", - "type": "string" - }, - { - "additionalProperties": false, - "properties": { - "headers": { - "additionalProperties": { - "type": "string" - }, - "type": "object" - }, - "metadata": { - "additionalProperties": {}, - "type": "object" - }, - "selectors": { - "additionalProperties": { - "type": "string" - }, - "type": "object" - }, - "timeout": { - "maximum": 30000, - "minimum": 1000, - "type": "number" - }, - "url": { - "format": "uri", - "type": "string" - } - }, - "required": [ - "url" - ], - "type": "object" - } -]New value: +[ + { + "format": "uri", + "type": "string" + }, + { + "properties": { + "headers": { + "additionalProperties": { + "type": "string" + }, + "propertyNames": { + "type": "string" + }, + "type": "object" + }, + "metadata": { + "additionalProperties": {}, + "propertyNames": { + "type": "string" + }, + "type": "object" + }, + "selectors": { + "additionalProperties": { + "type": "string" + }, + "propertyNames": { + "type": "string" + }, + "type": "object" + }, + "timeout": { + "maximum": 30000, + "minimum": 1000, + "type": "number" + }, + "url": { + "format": "uri", + "type": "string" + } + }, + "required": [ + "url" + ], + "type": "object" + } +] - removed
Input schema / properties / webhook / additionalPropertiesRemoved value: -false - added
Input schema / properties / webhook / properties / headers / propertyNamesAdded value: +{ + "type": "string" +}
- Changed
crawl_deep28 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - removed
Input schema / properties / domain_filter / additionalPropertiesRemoved value: -false - added
Input schema / properties / domain_filter / properties / blacklist / itemsAdded value: +{} - added
Input schema / properties / domain_filter / properties / domain_rules / propertyNamesAdded value: +{ + "type": "string" +} - added
Input schema / properties / domain_filter / properties / whitelist / itemsAdded value: +{} - removed
Input schema / properties / link_analysis_options / additionalPropertiesRemoved value: -false - added
Input schema / properties / max_inline_charsAdded value: +{ + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" +} - added
Input schema / properties / redact_piiAdded value: +{ + "anyOf": [ + { + "type": "boolean" + }, + { + "properties": { + "entities": { + "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected", + "items": { + "type": "string" + }, + "type": "array" + }, + "mode": { + "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "replace_style": { + "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value", + "enum": [ + "tag", + "mask", + "remove" + ], + "type": "string" + } + }, + "type": "object" + } + ], + "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off" +} - removed
Input schema / properties / session / additionalPropertiesRemoved value: -false - added
Input schema / properties / session / properties / headers / propertyNamesAdded value: +{ + "type": "string" +} - removed
Input schema / properties / session / properties / initialRequest / additionalPropertiesRemoved value: -false - added
Input schema / properties / session / properties / initialRequest / properties / headers / propertyNamesAdded value: +{ + "type": "string" +} - changed
Output schema / properties / _cost / additionalPropertiesPrevious value: -trueNew value: +{} - added
Output schema / properties / expires_atAdded value: +{ + "description": "When the stored result is dropped (ISO 8601)", + "type": "string" +} - added
Output schema / properties / previewAdded value: +{ + "description": "The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON)", + "type": "string" +} - added
Output schema / properties / redactionAdded value: +{ + "additionalProperties": {}, + "description": "Present when redact_pii was set: what was redacted from the text of this result", + "properties": { + "count": { + "description": "Total spans replaced", + "type": "number" + }, + "entities": { + "additionalProperties": { + "type": "number" + }, + "description": "How many spans were replaced, by entity class; a class with no hits is omitted", + "propertyNames": { + "type": "string" + }, + "type": "object" + }, + "mode": { + "description": "\"fast\" is the free regex pass; \"model\" added an Ollama NER pass for PERSON and LOCATION", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "model_ran": { + "description": "mode \"model\" only: whether a model actually answered. False means no LLM route existed and the model surcharge was not charged", + "type": "boolean" + } + }, + "type": "object" +} - added
Output schema / properties / result_handleAdded value: +{ + "description": "Handle for read_result; the full result is kept 1 hour", + "type": "string" +} - changed
Output schema / properties / results / items / additionalPropertiesPrevious value: -trueNew value: +{} - changed
Output schema / properties / session / additionalPropertiesPrevious value: -trueNew value: +{} - changed
Output schema / properties / site_structure / additionalPropertiesPrevious value: -trueNew value: +{} - added
Output schema / properties / site_structure / properties / depth_distribution / propertyNamesAdded value: +{ + "type": "string" +} - added
Output schema / properties / site_structure / properties / file_types / propertyNamesAdded value: +{ + "type": "string" +} - added
Output schema / properties / site_structure / properties / path_depth_distribution / propertyNamesAdded value: +{ + "type": "string" +} - added
Output schema / properties / site_structure / properties / path_patterns / propertyNamesAdded value: +{ + "type": "string" +} - added
Output schema / properties / total_charsAdded value: +{ + "description": "Length of the full view in characters", + "type": "number" +} - added
Output schema / properties / truncatedAdded value: +{ + "description": "True when the inline result is a preview", + "type": "boolean" +} - added
Output schema / properties / viewAdded value: +{ + "description": "Whether preview and read_result offsets index a text field or the pretty-printed JSON", + "enum": [ + "text", + "json" + ], + "type": "string" +} - added
Output schema / properties / view_pathAdded value: +{ + "description": "Dotted path of the text field the view was cut from; null for the JSON view", + "type": [ + "string", + "null" + ] +}
- Changed
deep_research9 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - removed
Input schema / properties / llmConfig / additionalPropertiesRemoved value: -false - removed
Input schema / properties / llmConfig / properties / anthropic / additionalPropertiesRemoved value: -false - removed
Input schema / properties / llmConfig / properties / ollama / additionalPropertiesRemoved value: -false - removed
Input schema / properties / llmConfig / properties / openai / additionalPropertiesRemoved value: -false - added
Input schema / properties / max_inline_charsAdded value: +{ + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" +} - removed
Input schema / properties / queryExpansion / additionalPropertiesRemoved value: -false - removed
Input schema / properties / webhook / additionalPropertiesRemoved value: -false - added
Input schema / properties / webhook / properties / headers / propertyNamesAdded value: +{ + "type": "string" +}
- Changed
extract_content4 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - added
Input schema / properties / max_inline_charsAdded value: +{ + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" +} - changed
Input schema / properties / options / additionalPropertiesPrevious value: -trueNew value: +{} - added
Input schema / properties / redact_piiAdded value: +{ + "anyOf": [ + { + "type": "boolean" + }, + { + "properties": { + "entities": { + "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected", + "items": { + "type": "string" + }, + "type": "array" + }, + "mode": { + "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "replace_style": { + "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value", + "enum": [ + "tag", + "mask", + "remove" + ], + "type": "string" + } + }, + "type": "object" + } + ], + "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off" +}
- Changed
extract_embedded_state2 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - added
Input schema / properties / max_inline_charsAdded value: +{ + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" +}
- Changed
extract_links1 field changed- removed
Input schema / additionalPropertiesRemoved value: -false
- Changed
extract_metadata1 field changed- removed
Input schema / additionalPropertiesRemoved value: -false
- Changed
extract_structured11 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - removed
Input schema / properties / llmConfig / additionalPropertiesRemoved value: -false - removed
Input schema / properties / schema / additionalPropertiesRemoved value: -false - added
Input schema / properties / schema / properties / properties / propertyNamesAdded value: +{ + "type": "string" +} - added
Input schema / properties / selectorHints / propertyNamesAdded value: +{ + "type": "string" +} - changed
Output schema / properties / _cost / additionalPropertiesPrevious value: -trueNew value: +{} - added
Output schema / properties / data / propertyNamesAdded value: +{ + "type": "string" +} - changed
Output schema / properties / provenance / additionalPropertiesPrevious value: -trueNew value: +{} - changed
Output schema / properties / provenance / properties / unverified / items / additionalPropertiesPrevious value: -trueNew value: +{} - added
Output schema / properties / schema_used / propertyNamesAdded value: +{ + "type": "string" +} - changed
Output schema / properties / validation / additionalPropertiesPrevious value: -trueNew value: +{}
- Changed
extract_text2 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - added
Input schema / properties / redact_piiAdded value: +{ + "anyOf": [ + { + "type": "boolean" + }, + { + "properties": { + "entities": { + "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected", + "items": { + "type": "string" + }, + "type": "array" + }, + "mode": { + "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "replace_style": { + "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value", + "enum": [ + "tag", + "mask", + "remove" + ], + "type": "string" + } + }, + "type": "object" + } + ], + "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off" +}
- Changed
extract_with_llm2 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - added
Input schema / properties / schema / propertyNamesAdded value: +{ + "type": "string" +}
- Changed
fetch_url3 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - added
Input schema / properties / headers / propertyNamesAdded value: +{ + "type": "string" +} - added
Input schema / properties / max_inline_charsAdded value: +{ + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" +}
- Changed
generate_llms_txt4 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - removed
Input schema / properties / analysisOptions / additionalPropertiesRemoved value: -false - removed
Input schema / properties / outputOptions / additionalPropertiesRemoved value: -false - added
Input schema / properties / outputOptions / properties / contactEmail / patternAdded value: +"^(?!\\.)(?!.*\\.\\.)([A-Za-z0-9_'+\\-\\.]*)[A-Za-z0-9_+-]@([A-Za-z0-9][A-Za-z0-9\\-]*\\.)+[A-Za-z]{2,}$"
- Changed
get_batch_results1 field changed- removed
Input schema / additionalPropertiesRemoved value: -false
- Changed
localization11 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - removed
Input schema / properties / browserOptions / additionalPropertiesRemoved value: -false - added
Input schema / properties / browserOptions / properties / extraHTTPHeaders / propertyNamesAdded value: +{ + "type": "string" +} - added
Input schema / properties / customHeaders / propertyNamesAdded value: +{ + "type": "string" +} - removed
Input schema / properties / geoLocation / additionalPropertiesRemoved value: -false - removed
Input schema / properties / proxySettings / additionalPropertiesRemoved value: -false - removed
Input schema / properties / proxySettings / properties / fallback / additionalPropertiesRemoved value: -false - removed
Input schema / properties / proxySettings / properties / rotation / additionalPropertiesRemoved value: -false - removed
Input schema / properties / response / additionalPropertiesRemoved value: -false - removed
Input schema / properties / searchParams / additionalPropertiesRemoved value: -false - added
Input schema / properties / searchParams / properties / headers / propertyNamesAdded value: +{ + "type": "string" +}
- Changed
map_site12 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - removed
Input schema / properties / domain_filter / additionalPropertiesRemoved value: -false - changed
Output schema / properties / _cost / additionalPropertiesPrevious value: -trueNew value: +{} - added
Output schema / properties / metadata / propertyNamesAdded value: +{ + "type": "string" +} - changed
Output schema / properties / ranked_urls / items / additionalPropertiesPrevious value: -trueNew value: +{} - changed
Output schema / properties / site_map / additionalPropertiesPrevious value: -trueNew value: +{} - added
Output schema / properties / site_map / properties / depth_levels / propertyNamesAdded value: +{ + "type": "string" +} - added
Output schema / properties / site_map / properties / sections / propertyNamesAdded value: +{ + "type": "string" +} - changed
Output schema / properties / statistics / additionalPropertiesPrevious value: -trueNew value: +{} - added
Output schema / properties / statistics / properties / file_extensions / propertyNamesAdded value: +{ + "type": "string" +} - changed
Output schema / properties / statistics / properties / url_lengths / additionalPropertiesPrevious value: -trueNew value: +{} - changed
Output schema / properties / urls / anyOfPrevious value: -[ - { - "items": { - "type": "string" - }, - "type": "array" - }, - { - "additionalProperties": { - "items": { - "type": "string" - }, - "type": "array" - }, - "type": "object" - } -]New value: +[ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "additionalProperties": { + "items": { + "type": "string" + }, + "type": "array" + }, + "propertyNames": { + "type": "string" + }, + "type": "object" + } +]
- Changed
process_document4 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - added
Input schema / properties / max_inline_charsAdded value: +{ + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" +} - changed
Input schema / properties / options / additionalPropertiesPrevious value: -trueNew value: +{} - added
Input schema / properties / redact_piiAdded value: +{ + "anyOf": [ + { + "type": "boolean" + }, + { + "properties": { + "entities": { + "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected", + "items": { + "type": "string" + }, + "type": "array" + }, + "mode": { + "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "replace_style": { + "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value", + "enum": [ + "tag", + "mask", + "remove" + ], + "type": "string" + } + }, + "type": "object" + } + ], + "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off" +}
- Added
read_result - Changed
reddit_search4 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - changed
Output schema / properties / _cost / additionalPropertiesPrevious value: -trueNew value: +{} - changed
Output schema / properties / post / anyOfPrevious value: -[ - { - "$ref": "#/properties/results/items/anyOf/0" - }, - { - "type": "null" - } -]New value: +[ + { + "additionalProperties": {}, + "properties": { + "author": { + "type": [ + "string", + "null" + ] + }, + "created_iso": { + "type": [ + "string", + "null" + ] + }, + "created_utc": { + "type": [ + "number", + "null" + ] + }, + "id": { + "type": [ + "string", + "null" + ] + }, + "num_comments": { + "type": [ + "number", + "null" + ] + }, + "permalink": { + "description": "Full reddit.com URL of the post", + "type": [ + "string", + "null" + ] + }, + "score": { + "type": [ + "number", + "null" + ] + }, + "selftext": { + "type": [ + "string", + "null" + ] + }, + "selftext_truncated": { + "type": "boolean" + }, + "subreddit": { + "type": [ + "string", + "null" + ] + }, + "title": { + "type": [ + "string", + "null" + ] + }, + "url": { + "type": [ + "string", + "null" + ] + } + }, + "type": "object" + }, + { + "type": "null" + } +] - changed
Output schema / properties / results / items / anyOfPrevious value: -[ - { - "additionalProperties": true, - "properties": { - "author": { - "type": [ - "string", - "null" - ] - }, - "created_iso": { - "type": [ - "string", - "null" - ] - }, - "created_utc": { - "type": [ - "number", - "null" - ] - }, - "id": { - "type": [ - "string", - "null" - ] - }, - "num_comments": { - "type": [ - "number", - "null" - ] - }, - "permalink": { - "description": "Full reddit.com URL of the post", - "type": [ - "string", - "null" - ] - }, - "score": { - "type": [ - "number", - "null" - ] - }, - "selftext": { - "type": [ - "string", - "null" - ] - }, - "selftext_truncated": { - "type": "boolean" - }, - "subreddit": { - "type": [ - "string", - "null" - ] - }, - "title": { - "type": [ - "string", - "null" - ] - }, - "url": { - "type": [ - "string", - "null" - ] - } - }, - "type": "object" - }, - { - "additionalProperties": true, - "properties": { - "author": { - "type": [ - "string", - "null" - ] - }, - "body": { - "type": [ - "string", - "null" - ] - }, - "body_truncated": { - "type": "boolean" - }, - "created_iso": { - "type": [ - "string", - "null" - ] - }, - "created_utc": { - "type": [ - "number", - "null" - ] - }, - "id": { - "type": [ - "string", - "null" - ] - }, - "link_id": { - "type": [ - "string", - "null" - ] - }, - "parent_id": { - "type": [ - "string", - "null" - ] - }, - "permalink": { - "type": [ - "string", - "null" - ] - }, - "score": { - "type": [ - "number", - "null" - ] - }, - "subreddit": { - "type": [ - "string", - "null" - ] - } - }, - "type": "object" - } -]New value: +[ + { + "additionalProperties": {}, + "properties": { + "author": { + "type": [ + "string", + "null" + ] + }, + "created_iso": { + "type": [ + "string", + "null" + ] + }, + "created_utc": { + "type": [ + "number", + "null" + ] + }, + "id": { + "type": [ + "string", + "null" + ] + }, + "num_comments": { + "type": [ + "number", + "null" + ] + }, + "permalink": { + "description": "Full reddit.com URL of the post", + "type": [ + "string", + "null" + ] + }, + "score": { + "type": [ + "number", + "null" + ] + }, + "selftext": { + "type": [ + "string", + "null" + ] + }, + "selftext_truncated": { + "type": "boolean" + }, + "subreddit": { + "type": [ + "string", + "null" + ] + }, + "title": { + "type": [ + "string", + "null" + ] + }, + "url": { + "type": [ + "string", + "null" + ] + } + }, + "type": "object" + }, + { + "additionalProperties": {}, + "properties": { + "author": { + "type": [ + "string", + "null" + ] + }, + "body": { + "type": [ + "string", + "null" + ] + }, + "body_truncated": { + "type": "boolean" + }, + "created_iso": { + "type": [ + "string", + "null" + ] + }, + "created_utc": { + "type": [ + "number", + "null" + ] + }, + "id": { + "type": [ + "string", + "null" + ] + }, + "link_id": { + "type": [ + "string", + "null" + ] + }, + "parent_id": { + "type": [ + "string", + "null" + ] + }, + "permalink": { + "type": [ + "string", + "null" + ] + }, + "score": { + "type": [ + "number", + "null" + ] + }, + "subreddit": { + "type": [ + "string", + "null" + ] + } + }, + "type": "object" + } +]
- Changed
scrape33 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - removed
Input schema / properties / brandingOptions / additionalPropertiesRemoved value: -false - added
Input schema / properties / escalateAdded value: +{ + "default": false, + "description": "When the plain fetch comes back blocked (403/429/challenge page/empty shell), retry once in the stealth browser and return its content instead of the block. Projected at 2+5; the actual charge stays at the base price when the plain fetch succeeded. Default: false", + "type": "boolean" +} - added
Input schema / properties / escalate_engineAdded value: +{ + "default": "playwright", + "description": "Stealth engine for the escalated retry (default: \"playwright\")", + "enum": [ + "playwright", + "camoufox" + ], + "type": "string" +} - changed
Input schema / properties / formats / items / anyOfPrevious value: -[ - { - "enum": [ - "markdown", - "html", - "rawHtml", - "text", - "links", - "metadata", - "screenshot", - "branding" - ], - "type": "string" - }, - { - "additionalProperties": false, - "properties": { - "prompt": { - "description": "Extraction instruction for the LLM", - "type": "string" - }, - "schema": { - "additionalProperties": {}, - "description": "JSON schema for extraction", - "type": "object" - }, - "type": { - "const": "json", - "type": "string" - } - }, - "required": [ - "type" - ], - "type": "object" - } -]New value: +[ + { + "enum": [ + "markdown", + "html", + "rawHtml", + "text", + "links", + "metadata", + "screenshot", + "branding" + ], + "type": "string" + }, + { + "properties": { + "prompt": { + "description": "Extraction instruction for the LLM", + "type": "string" + }, + "schema": { + "additionalProperties": {}, + "description": "JSON schema for extraction", + "propertyNames": { + "type": "string" + }, + "type": "object" + }, + "type": { + "const": "json", + "type": "string" + } + }, + "required": [ + "type" + ], + "type": "object" + }, + { + "properties": { + "max_highlights": { + "default": 10, + "description": "How many units to return (default 10)", + "maximum": 50, + "minimum": 1, + "type": "integer" + }, + "mode": { + "default": "extractive", + "description": "\"extractive\" (default) returns verbatim page text, no model; \"model\" adds an LLM step (+3 credits)", + "enum": [ + "extractive", + "model" + ], + "type": "string" + }, + "query": { + "description": "What to look for; the matching sentences, table rows and code blocks come back verbatim with offsets into the markdown", + "maxLength": 500, + "minLength": 1, + "type": "string" + }, + "type": { + "const": "highlights", + "type": "string" + } + }, + "required": [ + "type", + "query" + ], + "type": "object" + }, + { + "properties": { + "mode": { + "default": "extractive", + "description": "\"extractive\" (default) returns verbatim page text, no model; \"model\" adds an LLM step (+3 credits)", + "enum": [ + "extractive", + "model" + ], + "type": "string" + }, + "question": { + "description": "The question to answer from the page; the evidence units come back verbatim with offsets", + "maxLength": 500, + "minLength": 1, + "type": "string" + }, + "type": { + "const": "question", + "type": "string" + } + }, + "required": [ + "type", + "question" + ], + "type": "object" + } +] - added
Input schema / properties / max_inline_charsAdded value: +{ + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" +} - added
Input schema / properties / redact_piiAdded value: +{ + "anyOf": [ + { + "type": "boolean" + }, + { + "properties": { + "entities": { + "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected", + "items": { + "type": "string" + }, + "type": "array" + }, + "mode": { + "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "replace_style": { + "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value", + "enum": [ + "tag", + "mask", + "remove" + ], + "type": "string" + } + }, + "type": "object" + } + ], + "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off" +} - removed
Input schema / properties / screenshotOptions / additionalPropertiesRemoved value: -false - changed
Output schema / properties / _cost / additionalPropertiesPrevious value: -trueNew value: +{} - added
Output schema / properties / blockedAdded value: +{ + "additionalProperties": {}, + "description": "Present when a bot-defence vendor served a challenge page; the fallback hint names the tool to try next", + "properties": { + "evidence": { + "type": "string" + }, + "vendor": { + "type": "string" + } + }, + "type": "object" +} - changed
Output schema / properties / content / additionalPropertiesPrevious value: -trueNew value: +{} - added
Output schema / properties / content / properties / answerAdded value: +{ + "additionalProperties": {}, + "description": "Result of the {type:\"question\"} format", + "properties": { + "evidence": { + "description": "The units the answer rests on, verbatim with offsets", + "items": { + "additionalProperties": {}, + "properties": { + "kind": { + "enum": [ + "sentence", + "table_row", + "code_block" + ], + "type": "string" + }, + "length": { + "type": "number" + }, + "offset": { + "description": "JS string index into the markdown format of this call", + "type": "number" + }, + "score": { + "description": "BM25 relevance to the query, higher is better", + "type": "number" + }, + "text": { + "description": "Verbatim page text: markdown.slice(offset, offset + length) === text", + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + }, + "grounded": { + "description": "True when every number and proper noun in text appears in the evidence or the question; always true in extractive mode", + "type": "boolean" + }, + "text": { + "description": "Extractive mode: the evidence texts joined; model mode: the model's answer", + "type": "string" + } + }, + "type": "object" +} - added
Output schema / properties / content / properties / branding / propertyNamesAdded value: +{ + "type": "string" +} - added
Output schema / properties / content / properties / highlightsAdded value: +{ + "description": "Result of the {type:\"highlights\"} format: the units matching the query, best first, verbatim with offsets", + "items": { + "additionalProperties": {}, + "properties": { + "kind": { + "enum": [ + "sentence", + "table_row", + "code_block" + ], + "type": "string" + }, + "length": { + "type": "number" + }, + "offset": { + "description": "JS string index into the markdown format of this call", + "type": "number" + }, + "score": { + "description": "BM25 relevance to the query, higher is better", + "type": "number" + }, + "text": { + "description": "Verbatim page text: markdown.slice(offset, offset + length) === text", + "type": "string" + } + }, + "type": "object" + }, + "type": "array" +} - changed
Output schema / properties / content / properties / links / additionalPropertiesPrevious value: -trueNew value: +{} - changed
Output schema / properties / content / properties / links / properties / links / items / additionalPropertiesPrevious value: -trueNew value: +{} - changed
Output schema / properties / content / properties / metadata / additionalPropertiesPrevious value: -trueNew value: +{} - added
Output schema / properties / content / properties / metadata / properties / og_tags / propertyNamesAdded value: +{ + "type": "string" +} - added
Output schema / properties / content / properties / metadata / properties / twitter_tags / propertyNamesAdded value: +{ + "type": "string" +} - changed
Output schema / properties / content / properties / screenshots / items / additionalPropertiesPrevious value: -trueNew value: +{} - added
Output schema / properties / errorAdded value: +{ + "description": "Why success is false: a challenge page, an empty shell or an error placeholder was served instead of the content", + "type": "string" +} - added
Output schema / properties / escalatedAdded value: +{ + "description": "Present only when escalate:true was passed: whether the blocked plain fetch was retried in the stealth browser. False means the plain fetch sufficed and the call is charged at the base price", + "type": "boolean" +} - added
Output schema / properties / expires_atAdded value: +{ + "description": "When the stored result is dropped (ISO 8601)", + "type": "string" +} - added
Output schema / properties / previewAdded value: +{ + "description": "The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON)", + "type": "string" +} - added
Output schema / properties / redactionAdded value: +{ + "additionalProperties": {}, + "description": "Present when redact_pii was set: what was redacted from the text of this result", + "properties": { + "count": { + "description": "Total spans replaced", + "type": "number" + }, + "entities": { + "additionalProperties": { + "type": "number" + }, + "description": "How many spans were replaced, by entity class; a class with no hits is omitted", + "propertyNames": { + "type": "string" + }, + "type": "object" + }, + "mode": { + "description": "\"fast\" is the free regex pass; \"model\" added an Ollama NER pass for PERSON and LOCATION", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "model_ran": { + "description": "mode \"model\" only: whether a model actually answered. False means no LLM route existed and the model surcharge was not charged", + "type": "boolean" + } + }, + "type": "object" +} - added
Output schema / properties / result_handleAdded value: +{ + "description": "Handle for read_result; the full result is kept 1 hour", + "type": "string" +} - added
Output schema / properties / statusAdded value: +{ + "description": "HTTP status of the fetch; present when success is false", + "type": "number" +} - added
Output schema / properties / stealthAdded value: +{ + "additionalProperties": {}, + "description": "Present when escalated is true: the stealth engine that ran, and the bot-defence vendor the plain fetch hit (null when the block named none)", + "properties": { + "engine": { + "type": "string" + }, + "vendor_detected": { + "type": [ + "string", + "null" + ] + } + }, + "type": "object" +} - added
Output schema / properties / titleAdded value: +{ + "description": "Document title; present when success is false", + "type": "string" +} - added
Output schema / properties / total_charsAdded value: +{ + "description": "Length of the full view in characters", + "type": "number" +} - added
Output schema / properties / truncatedAdded value: +{ + "description": "True when the inline result is a preview", + "type": "boolean" +} - added
Output schema / properties / viewAdded value: +{ + "description": "Whether preview and read_result offsets index a text field or the pretty-printed JSON", + "enum": [ + "text", + "json" + ], + "type": "string" +} - added
Output schema / properties / view_pathAdded value: +{ + "description": "Dotted path of the text field the view was cut from; null for the JSON view", + "type": [ + "string", + "null" + ] +}
- Changed
scrape_structured3 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - added
Input schema / properties / max_results / maximumAdded value: +9007199254740991 - added
Input schema / properties / selectors / propertyNamesAdded value: +{ + "type": "string" +}
- Changed
scrape_template2 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - added
Input schema / properties / params / propertyNamesAdded value: +{ + "type": "string" +}
- Changed
scrape_with_actions11 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - removed
Input schema / properties / actions / items / additionalPropertiesRemoved value: -false - added
Input schema / properties / actions / items / properties / args / itemsAdded value: +{} - removed
Input schema / properties / actions / items / properties / position / additionalPropertiesRemoved value: -false - removed
Input schema / properties / browserOptions / additionalPropertiesRemoved value: -false - removed
Input schema / properties / extractionOptions / additionalPropertiesRemoved value: -false - added
Input schema / properties / extractionOptions / properties / selectors / propertyNamesAdded value: +{ + "type": "string" +} - removed
Input schema / properties / formAutoFill / additionalPropertiesRemoved value: -false - removed
Input schema / properties / formAutoFill / properties / fields / items / additionalPropertiesRemoved value: -false - added
Input schema / properties / max_inline_charsAdded value: +{ + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" +} - added
Input schema / properties / redact_piiAdded value: +{ + "anyOf": [ + { + "type": "boolean" + }, + { + "properties": { + "entities": { + "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected", + "items": { + "type": "string" + }, + "type": "array" + }, + "mode": { + "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "replace_style": { + "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value", + "enum": [ + "tag", + "mask", + "remove" + ], + "type": "string" + } + }, + "type": "object" + } + ], + "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off" +}
- Changed
search_web25 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - removed
Input schema / properties / deduplication_thresholds / additionalPropertiesRemoved value: -false - removed
Input schema / properties / expansion_options / additionalPropertiesRemoved value: -false - removed
Input schema / properties / localization / additionalPropertiesRemoved value: -false - removed
Input schema / properties / localization / properties / customLocation / additionalPropertiesRemoved value: -false - added
Input schema / properties / queriesAdded value: +{ + "description": "Run 1-10 searches in one call instead of 10 round-trips; every other parameter applies to each. Results come back in results_by_query, one entry per query, in order. Costs 5 per query. Use this OR query, not both", + "items": { + "minLength": 1, + "type": "string" + }, + "maxItems": 10, + "minItems": 1, + "type": "array" +} - changed
Input schema / properties / query / descriptionPrevious value: -"Search query string"New value: +"Search query string. Use this OR queries, not both" - removed
Input schema / properties / ranking_weights / additionalPropertiesRemoved value: -false - added
Input schema / properties / redact_piiAdded value: +{ + "anyOf": [ + { + "type": "boolean" + }, + { + "properties": { + "entities": { + "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected", + "items": { + "type": "string" + }, + "type": "array" + }, + "mode": { + "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "replace_style": { + "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value", + "enum": [ + "tag", + "mask", + "remove" + ], + "type": "string" + } + }, + "type": "object" + } + ], + "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off" +} - removed
Input schema / requiredRemoved value: -[ - "query" -] - changed
Output schema / properties / _cost / additionalPropertiesPrevious value: -trueNew value: +{} - added
Output schema / properties / countAdded value: +{ + "description": "Batch form: how many queries ran", + "type": "number" +} - changed
Output schema / properties / localization / anyOfPrevious value: -[ - { - "additionalProperties": true, - "properties": { - "applied": { - "type": "boolean" - }, - "countryCode": { - "type": "string" - }, - "geoTargeting": { - "type": "boolean" - }, - "language": { - "type": "string" - }, - "searchDomain": { - "type": "string" - } - }, - "type": "object" - }, - { - "type": "null" - } -]New value: +[ + { + "additionalProperties": {}, + "properties": { + "applied": { + "type": "boolean" + }, + "countryCode": { + "type": "string" + }, + "geoTargeting": { + "type": "boolean" + }, + "language": { + "type": "string" + }, + "searchDomain": { + "type": "string" + } + }, + "type": "object" + }, + { + "type": "null" + } +] - changed
Output schema / properties / processing / additionalPropertiesPrevious value: -trueNew value: +{} - changed
Output schema / properties / processing / properties / deduplication / anyOfPrevious value: -[ - { - "additionalProperties": {}, - "type": "object" - }, - { - "type": "null" - } -]New value: +[ + { + "additionalProperties": {}, + "propertyNames": { + "type": "string" + }, + "type": "object" + }, + { + "type": "null" + } +] - changed
Output schema / properties / processing / properties / query_expansion / anyOfPrevious value: -[ - { - "additionalProperties": {}, - "type": "object" - }, - { - "type": "null" - } -]New value: +[ + { + "additionalProperties": {}, + "propertyNames": { + "type": "string" + }, + "type": "object" + }, + { + "type": "null" + } +] - changed
Output schema / properties / processing / properties / ranking / anyOfPrevious value: -[ - { - "additionalProperties": {}, - "type": "object" - }, - { - "type": "null" - } -]New value: +[ + { + "additionalProperties": {}, + "propertyNames": { + "type": "string" + }, + "type": "object" + }, + { + "type": "null" + } +] - changed
Output schema / properties / provider / additionalPropertiesPrevious value: -trueNew value: +{} - added
Output schema / properties / provider / properties / capabilities / propertyNamesAdded value: +{ + "type": "string" +} - added
Output schema / properties / queriesAdded value: +{ + "description": "Batch form: the queries that ran, in order", + "items": { + "type": "string" + }, + "type": "array" +} - added
Output schema / properties / redactionAdded value: +{ + "additionalProperties": {}, + "description": "Present when redact_pii was set: what was redacted from the text of this result", + "properties": { + "count": { + "description": "Total spans replaced", + "type": "number" + }, + "entities": { + "additionalProperties": { + "type": "number" + }, + "description": "How many spans were replaced, by entity class; a class with no hits is omitted", + "propertyNames": { + "type": "string" + }, + "type": "object" + }, + "mode": { + "description": "\"fast\" is the free regex pass; \"model\" added an Ollama NER pass for PERSON and LOCATION", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "model_ran": { + "description": "mode \"model\" only: whether a model actually answered. False means no LLM route existed and the model surcharge was not charged", + "type": "boolean" + } + }, + "type": "object" +} - changed
Output schema / properties / results / items / additionalPropertiesPrevious value: -trueNew value: +{} - added
Output schema / properties / results / items / properties / metadata / propertyNamesAdded value: +{ + "type": "string" +} - added
Output schema / properties / results / items / properties / pagemap / propertyNamesAdded value: +{ + "type": "string" +} - added
Output schema / properties / results_by_queryAdded value: +{ + "description": "Batch form: one entry per query, in order", + "items": { + "additionalProperties": {}, + "properties": { + "error": { + "description": "Present when this query failed; the other queries in the batch are unaffected", + "type": "string" + }, + "query": { + "type": "string" + }, + "results": { + "items": { + "additionalProperties": {}, + "properties": { + "displayLink": { + "type": "string" + }, + "formattedUrl": { + "type": "string" + }, + "htmlSnippet": { + "type": "string" + }, + "link": { + "type": "string" + }, + "metadata": { + "additionalProperties": {}, + "propertyNames": { + "type": "string" + }, + "type": "object" + }, + "pagemap": { + "additionalProperties": {}, + "propertyNames": { + "type": "string" + }, + "type": "object" + }, + "snippet": { + "type": "string" + }, + "title": { + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + } + }, + "type": "object" + }, + "type": "array" +}
- Changed
serp_rank7 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - changed
Output schema / properties / _cost / additionalPropertiesPrevious value: -trueNew value: +{} - changed
Output schema / properties / allPositions / items / additionalPropertiesPrevious value: -trueNew value: +{} - removed
Output schema / properties / results / items / $refRemoved value: -"#/properties/allPositions/items" - added
Output schema / properties / results / items / additionalPropertiesAdded value: +{} - added
Output schema / properties / results / items / propertiesAdded value: +{ + "domain": { + "type": "string" + }, + "position": { + "type": [ + "number", + "null" + ] + }, + "rankAbsolute": { + "type": [ + "number", + "null" + ] + }, + "snippet": { + "type": [ + "string", + "null" + ] + }, + "title": { + "type": [ + "string", + "null" + ] + }, + "url": { + "type": [ + "string", + "null" + ] + } +} - added
Output schema / properties / results / items / typeAdded value: +"object"
- Changed
stealth_mode8 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - added
Input schema / properties / max_inline_charsAdded value: +{ + "description": "Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS)", + "maximum": 10000000, + "minimum": 1000, + "type": "integer" +} - added
Input schema / properties / redact_piiAdded value: +{ + "anyOf": [ + { + "type": "boolean" + }, + { + "properties": { + "entities": { + "description": "Which classes to redact, case-insensitive: EMAIL, PHONE, FINANCIAL, SECRET, plus PERSON and LOCATION when mode is \"model\". Omitted or empty means all four regex classes (and both model classes in \"model\" mode). An unknown name, or a model-only name without mode:\"model\", is rejected", + "items": { + "type": "string" + }, + "type": "array" + }, + "mode": { + "description": "\"fast\" (default) is regex only and free; \"model\" adds an Ollama NER pass for PERSON and LOCATION (+3 credits once per call)", + "enum": [ + "fast", + "model" + ], + "type": "string" + }, + "replace_style": { + "description": "\"tag\" (default) writes <EMAIL>, \"mask\" writes [REDACTED], \"remove\" deletes the value", + "enum": [ + "tag", + "mask", + "remove" + ], + "type": "string" + } + }, + "type": "object" + } + ], + "description": "Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off" +} - removed
Input schema / properties / stealthConfig / additionalPropertiesRemoved value: -false - removed
Input schema / properties / stealthConfig / properties / antiDetection / additionalPropertiesRemoved value: -false - removed
Input schema / properties / stealthConfig / properties / customViewport / additionalPropertiesRemoved value: -false - removed
Input schema / properties / stealthConfig / properties / fingerprinting / additionalPropertiesRemoved value: -false - removed
Input schema / properties / stealthConfig / properties / proxyRotation / additionalPropertiesRemoved value: -false
- Changed
summarize_content2 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - changed
Input schema / properties / options / additionalPropertiesPrevious value: -trueNew value: +{}
- Changed
track_changes14 fields changed- removed
Input schema / additionalPropertiesRemoved value: -false - removed
Input schema / properties / alertRuleOptions / additionalPropertiesRemoved value: -false - removed
Input schema / properties / dashboardOptions / additionalPropertiesRemoved value: -false - removed
Input schema / properties / exportOptions / additionalPropertiesRemoved value: -false - removed
Input schema / properties / monitoringOptions / additionalPropertiesRemoved value: -false - removed
Input schema / properties / notificationOptions / additionalPropertiesRemoved value: -false - removed
Input schema / properties / notificationOptions / properties / slack / additionalPropertiesRemoved value: -false - removed
Input schema / properties / notificationOptions / properties / webhook / additionalPropertiesRemoved value: -false - added
Input schema / properties / notificationOptions / properties / webhook / properties / headers / propertyNamesAdded value: +{ + "type": "string" +} - removed
Input schema / properties / queryOptions / additionalPropertiesRemoved value: -false - removed
Input schema / properties / scheduledMonitorOptions / additionalPropertiesRemoved value: -false - removed
Input schema / properties / storageOptions / additionalPropertiesRemoved value: -false - removed
Input schema / properties / trackingOptions / additionalPropertiesRemoved value: -false - removed
Input schema / properties / trackingOptions / properties / significanceThresholds / additionalPropertiesRemoved value: -false
3 tool updates
v5.6.6- Changed
deep_research3 fields changed- changed
Input schema / properties / llmConfig / descriptionPrevious value: -"LLM provider configuration for AI-powered analysis"New value: +"LLM provider configuration for AI-powered analysis. provider 'auto' (default) uses a configured cloud key if there is one, else the local Ollama (http://localhost:11434, no key); 'ollama' forces the local model; 'openai'/'anthropic' need the matching API key" - added
Input schema / properties / llmConfig / properties / ollamaAdded value: +{ + "additionalProperties": false, + "properties": { + "embeddingModel": { + "type": "string" + }, + "model": { + "type": "string" + } + }, + "type": "object" +} - changed
Input schema / properties / llmConfig / properties / provider / enumPrevious value: -[ - "auto", - "openai", - "anthropic" -]New value: +[ + "auto", + "openai", + "anthropic", + "ollama" +]
- Changed
reddit_search5 fields changed- changed
Input schema / properties / source / descriptionPrevious value: -"Backend: auto routes + falls back (default). reddit_api uses the official Reddit Data API — only when REDDIT_CLIENT_ID/REDDIT_CLIENT_SECRET are set; serves posts/thread, not comment search"New value: +"Backend: auto routes + falls back (default). web_discovery serves only unscoped keyword searches (web search finds the posts, the archive supplies the rows). reddit_api uses the official Reddit Data API — only when REDDIT_CLIENT_ID/REDDIT_CLIENT_SECRET are set; serves posts/thread, not comment search" - changed
Input schema / properties / source / enumPrevious value: -[ - "auto", - "arctic_shift", - "pullpush", - "reddit_api" -]New value: +[ + "auto", + "arctic_shift", + "pullpush", + "reddit_api", + "web_discovery" +] - added
Output schema / properties / discoveredAdded value: +{ + "description": "web_discovery: how many post ids the site-restricted web search surfaced before archive hydration", + "type": "number" +} - added
Output schema / properties / posts_searchedAdded value: +{ + "description": "web_discovery comments mode: how many discovered posts had their comments searched before limit was reached", + "type": "number" +} - added
Output schema / properties / window_appliedAdded value: +{ + "description": "arctic_shift comments mode: the after-window (\"7d\"/\"3d\"/\"1d\") the search was narrowed to after the full-history search timed out; absent when the caller set after or no narrowing was needed", + "type": "string" +}
- Changed
scrape_with_actions1 field changed- changed
Input schema / properties / extractionOptions / descriptionPrevious value: -"Content extraction options"New value: +"Content extraction options. selectors results are returned as content.json.extracted, so include \"json\" in formats when passing selectors — without it the extraction is not part of the response."
18 tool updates
v5.4.0- Changed
batch_scrape2 fields changed- added
Input schema / properties / respect_robotsAdded value: +{ + "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", + "type": "boolean" +} - added
Input schema / properties / user_agentAdded value: +{ + "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.", + "type": "string" +}
- Changed
crawl_deep2 fields changed- added
Output schema / properties / site_structure / properties / depth_distribution / descriptionAdded value: +"Pages per crawl depth (links from the start URL)" - added
Output schema / properties / site_structure / properties / path_depth_distributionAdded value: +{ + "additionalProperties": { + "type": "number" + }, + "description": "Pages per URL path-segment depth", + "type": "object" +}
- Changed
extract_content2 fields changed- added
Input schema / properties / respect_robotsAdded value: +{ + "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", + "type": "boolean" +} - added
Input schema / properties / user_agentAdded value: +{ + "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.", + "type": "string" +}
- Added
extract_embedded_state - Changed
extract_links2 fields changed- added
Input schema / properties / respect_robotsAdded value: +{ + "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", + "type": "boolean" +} - added
Input schema / properties / user_agentAdded value: +{ + "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.", + "type": "string" +}
- Changed
extract_metadata3 fields changed- added
Input schema / properties / json_ld_typesAdded value: +{ + "description": "Filter the returned JSON-LD to nodes of these schema.org types, e.g. [\"Product\",\"Offer\"]. Subtypes match their parent: \"Event\" returns MusicEvent, \"Offer\" returns AggregateOffer, \"ItemList\" returns BreadcrumbList. Nodes are found at any depth, including inside @graph and nested inside a parent node. When set, json_ld carries only the matching nodes instead of the raw dump, and json_ld_type_counts reports how many matched per requested type. Documented types: ItemList, Product, Offer, Event, JobPosting, RealEstateListing — any other schema.org type is matched exactly.", + "items": { + "type": "string" + }, + "type": "array" +} - added
Input schema / properties / respect_robotsAdded value: +{ + "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", + "type": "boolean" +} - added
Input schema / properties / user_agentAdded value: +{ + "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.", + "type": "string" +}
- Changed
extract_structured5 fields changed- added
Input schema / properties / respect_robotsAdded value: +{ + "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", + "type": "boolean" +} - added
Input schema / properties / user_agentAdded value: +{ + "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.", + "type": "string" +} - added
Input schema / properties / verify_numbersAdded value: +{ + "default": true, + "description": "Numeric provenance guard (default: true): every price or numeric value the LLM returns must appear literally in the page source, else it is returned as null with a reason in `provenance.unverified`. Set false to get the model's raw numbers back, including ones it derived (a count, a sum, a total) rather than read off the page.", + "type": "boolean" +} - added
Output schema / properties / provenanceAdded value: +{ + "additionalProperties": true, + "properties": { + "enabled": { + "description": "Whether the numeric provenance guard ran", + "type": "boolean" + }, + "nulled": { + "description": "Numeric values replaced with null because the source does not contain them", + "type": "number" + }, + "skipped": { + "description": "\"empty_source\" when there was nothing to check against", + "type": "string" + }, + "unverified": { + "items": { + "additionalProperties": true, + "properties": { + "path": { + "description": "Path to the field, e.g. configurations[2].price", + "type": "string" + }, + "reason": { + "description": "\"not_found_in_source\"", + "type": "string" + }, + "value": { + "description": "The value that was removed" + } + }, + "type": "object" + }, + "type": "array" + }, + "verified": { + "description": "Numeric values found literally in the page source", + "type": "number" + } + }, + "type": "object" +} - added
Output schema / properties / successAdded value: +{ + "description": "False when the extraction errored or a required field came back missing or empty", + "type": "boolean" +}
- Changed
extract_text2 fields changed- added
Input schema / properties / respect_robotsAdded value: +{ + "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", + "type": "boolean" +} - added
Input schema / properties / user_agentAdded value: +{ + "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.", + "type": "string" +}
- Changed
extract_with_llm3 fields changed- added
Input schema / properties / respect_robotsAdded value: +{ + "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", + "type": "boolean" +} - added
Input schema / properties / user_agentAdded value: +{ + "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.", + "type": "string" +} - added
Input schema / properties / verify_numbersAdded value: +{ + "default": true, + "description": "Numeric provenance guard (default: true): every price or numeric value the LLM returns must appear literally in the page source, else it is returned as null with a reason in `provenance.unverified`. Set false to get the model's raw numbers back, including ones it derived (a count, a sum, a total) rather than read off the page.", + "type": "boolean" +}
- Changed
fetch_url2 fields changed- added
Input schema / properties / respect_robotsAdded value: +{ + "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", + "type": "boolean" +} - added
Input schema / properties / user_agentAdded value: +{ + "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.", + "type": "string" +}
- Changed
map_site2 fields changed- added
Input schema / properties / respect_robotsAdded value: +{ + "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", + "type": "boolean" +} - added
Input schema / properties / user_agentAdded value: +{ + "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.", + "type": "string" +}
- Changed
process_document2 fields changed- added
Input schema / properties / respect_robotsAdded value: +{ + "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", + "type": "boolean" +} - added
Input schema / properties / user_agentAdded value: +{ + "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.", + "type": "string" +}
- Changed
scrape2 fields changed- added
Input schema / properties / respect_robotsAdded value: +{ + "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", + "type": "boolean" +} - added
Input schema / properties / user_agentAdded value: +{ + "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.", + "type": "string" +}
- Changed
scrape_structured4 fields changed- changed
Input schema / properties / max_results / descriptionPrevious value: -"Maximum number of matches to return per field when a selector matches multiple elements"New value: +"Maximum number of matches to return per field when a selector matches multiple elements, or the maximum number of rows when row_selector is set" - added
Input schema / properties / respect_robotsAdded value: +{ + "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", + "type": "boolean" +} - added
Input schema / properties / row_selectorAdded value: +{ + "description": "CSS selector for the repeating row/container element. When set, each field in selectors is matched inside each row and data is an array of row-aligned records ({field: value|null}) instead of parallel arrays", + "type": "string" +} - added
Input schema / properties / user_agentAdded value: +{ + "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.", + "type": "string" +}
- Changed
scrape_template5 fields changed- added
Input schema / properties / paramsAdded value: +{ + "additionalProperties": {}, + "description": "Parameters for a list connector, e.g. {company:\"stripe\"} for greenhouse-jobs or {store:\"www.allbirds.com\", collection:\"mens\"} for shopify-collection. Use template:\"list\" to see which templates take params", + "type": "object" +} - added
Input schema / properties / respect_robotsAdded value: +{ + "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", + "type": "boolean" +} - changed
Input schema / properties / template / descriptionPrevious value: -"Template ID (e.g. github-repo) or list to enumerate available templates"New value: +"Template ID (e.g. github-repo), \"auto\" to detect one from the url, or \"list\" to enumerate available templates" - changed
Input schema / properties / url / descriptionPrevious value: -"URL to scrape — required unless template is list"New value: +"URL to scrape — required unless template is list, or params drive a list connector" - added
Input schema / properties / user_agentAdded value: +{ + "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.", + "type": "string" +}
- Changed
scrape_with_actions7 fields changed- changed
Input schema / properties / actions / items / properties / type / enumPrevious value: -[ - "wait", - "click", - "type", - "press", - "scroll", - "screenshot", - "executeJavaScript" -]New value: +[ + "wait", + "click", + "type", + "press", + "scroll", + "screenshot", + "executeJavaScript", + "select", + "hover", + "navigate" +] - added
Input schema / properties / actions / items / properties / urlAdded value: +{ + "description": "navigate: URL to navigate to — goes through the same SSRF and robots.txt gate as the initial URL", + "format": "uri", + "type": "string" +} - added
Input schema / properties / actions / items / properties / valueAdded value: +{ + "description": "select: option to choose, matched by value or label", + "type": "string" +} - added
Input schema / properties / actions / items / properties / valuesAdded value: +{ + "description": "select: options to choose in a multi-select, matched by value or label", + "items": { + "type": "string" + }, + "type": "array" +} - added
Input schema / properties / actions / items / properties / waitUntilAdded value: +{ + "description": "navigate: when to consider navigation complete", + "enum": [ + "load", + "domcontentloaded", + "networkidle", + "commit" + ], + "type": "string" +} - added
Input schema / properties / browserOptions / properties / stealthAdded value: +{ + "default": false, + "description": "Run the action chain in the stealth browser (randomized fingerprint, WebRTC/canvas spoofing) instead of the standard browser pool. Renders JavaScript; it does not solve challenges.", + "type": "boolean" +} - added
Input schema / properties / respect_robotsAdded value: +{ + "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", + "type": "boolean" +}
- Changed
stealth_mode6 fields changed- added
Input schema / properties / formatsAdded value: +{ + "default": [ + "markdown" + ], + "description": "Formats to return from operation:\"scrape\" (default: [\"markdown\"]). \"screenshot\" returns a crawlforge://screenshot/{id} resource URI.", + "items": { + "enum": [ + "markdown", + "html", + "text", + "links", + "metadata", + "screenshot" + ], + "type": "string" + }, + "type": "array" +} - changed
Input schema / properties / operation / enumPrevious value: -[ - "configure", - "enable", - "disable", - "create_context", - "create_page", - "get_stats", - "cleanup" -]New value: +[ + "scrape", + "configure", + "enable", + "disable", + "create_context", + "create_page", + "get_stats", + "cleanup" +] - added
Input schema / properties / respect_robotsAdded value: +{ + "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", + "type": "boolean" +} - added
Input schema / properties / urlAdded value: +{ + "description": "URL to scrape — required for operation:\"scrape\"", + "format": "uri", + "type": "string" +} - added
Input schema / properties / verboseAdded value: +{ + "default": false, + "description": "Return the full generated fingerprint from create_context instead of a summary", + "type": "boolean" +} - added
Input schema / properties / wait_forAdded value: +{ + "description": "Extra wait after page load, in ms — for content that renders after DOMContentLoaded", + "maximum": 30000, + "minimum": 0, + "type": "number" +}
- Changed
track_changes2 fields changed- added
Input schema / properties / respect_robotsAdded value: +{ + "description": "Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default.", + "type": "boolean" +} - added
Input schema / properties / user_agentAdded value: +{ + "description": "Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.", + "type": "string" +}
3 tool updates
v5.1.0- Changed
crawl_deep2 fields changed- added
Output schema / properties / cachedAdded value: +{ + "description": "True when this response was replayed from an earlier crawl rather than crawled now; crawled_at gives its age", + "type": "boolean" +} - added
Output schema / properties / crawled_atAdded value: +{ + "description": "When the pages were actually fetched (ISO 8601)", + "type": "string" +}
- Changed
extract_structured1 field changed- changed
Output schema / properties / extraction_method / descriptionPrevious value: -"\"llm\" | \"css_fallback\" | \"none\""New value: +"\"llm\" | \"css_fallback\" | \"keyword_fallback\" | \"none\""
- Added
reddit_search
1 tool update
v5.0.5- Changed
serp_rank1 field changed- changed
Input schema / properties / depth / descriptionPrevious value: -"How many results to scan, 10-200 (100 = 1 page of cost)"New value: +"How many results to scan, 10-200 (default 20; DataForSEO bills ~$0.002 per 10 and gets slower the deeper it goes)"
27 tool updates
v5.0.4- Changed
agent1 field changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
- Changed
analyze_content2 fields changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema" - changed
Input schema / properties / options / additionalPropertiesPrevious value: -falseNew value: +true
- Changed
batch_scrape1 field changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
- Changed
crawl_deep2 fields changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema" - changed
Output schema / (root)Previous value: -nullNew value: +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "additionalProperties": false, + "properties": { + "_cost": { + "additionalProperties": true, + "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result", + "properties": { + "actual": { + "description": "Credits actually charged (0 in creator mode, half-rate on error)", + "type": "number" + }, + "projected": { + "description": "Credits projected for this call before execution", + "type": "number" + }, + "projection_note": { + "description": "Human-readable note about how the cost was projected", + "type": "string" + }, + "remaining_credits": { + "description": "Credits remaining on the account after this call, if known", + "type": [ + "number", + "null" + ] + } + }, + "type": "object" + }, + "crawl_depth": { + "type": "number" + }, + "domain_filter_config": { + "anyOf": [ + {}, + { + "type": "null" + } + ] + }, + "duration_ms": { + "type": "number" + }, + "error": { + "type": "string" + }, + "error_count": { + "type": "number" + }, + "errors": { + "items": {}, + "type": "array" + }, + "link_analysis": { + "anyOf": [ + {}, + { + "type": "null" + } + ] + }, + "pages_crawled": { + "type": "number" + }, + "pages_found": { + "type": "number" + }, + "pages_per_second": { + "type": "number" + }, + "results": { + "items": { + "additionalProperties": true, + "properties": { + "content": { + "type": "string" + }, + "content_length": { + "type": "number" + }, + "depth": { + "type": "number" + }, + "links_count": { + "type": "number" + }, + "metadata": {}, + "timestamp": { + "type": [ + "string", + "number" + ] + }, + "title": { + "type": "string" + }, + "truncated": { + "type": "boolean" + }, + "url": { + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + }, + "session": { + "additionalProperties": true, + "properties": { + "cookies_captured": { + "type": "number" + }, + "enabled": { + "type": "boolean" + } + }, + "type": "object" + }, + "site_structure": { + "additionalProperties": true, + "properties": { + "depth_distribution": { + "additionalProperties": { + "type": "number" + }, + "type": "object" + }, + "file_types": { + "additionalProperties": { + "type": "number" + }, + "type": "object" + }, + "path_patterns": { + "additionalProperties": { + "type": "number" + }, + "type": "object" + }, + "subdomains": { + "items": { + "type": "string" + }, + "type": "array" + }, + "total_pages": { + "type": "number" + } + }, + "type": "object" + }, + "stats": {}, + "success": { + "description": "False only when the crawl was cancelled via elicitation decline", + "type": "boolean" + }, + "url": { + "type": "string" + } + }, + "type": "object" +}
- Changed
deep_research1 field changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
- Changed
extract_content2 fields changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema" - changed
Input schema / properties / options / additionalPropertiesPrevious value: -falseNew value: +true
- Changed
extract_links1 field changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
- Changed
extract_metadata1 field changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
- Changed
extract_structured2 fields changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema" - changed
Output schema / (root)Previous value: -nullNew value: +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "additionalProperties": false, + "properties": { + "_cost": { + "additionalProperties": true, + "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result", + "properties": { + "actual": { + "description": "Credits actually charged (0 in creator mode, half-rate on error)", + "type": "number" + }, + "projected": { + "description": "Credits projected for this call before execution", + "type": "number" + }, + "projection_note": { + "description": "Human-readable note about how the cost was projected", + "type": "string" + }, + "remaining_credits": { + "description": "Credits remaining on the account after this call, if known", + "type": [ + "number", + "null" + ] + } + }, + "type": "object" + }, + "confidence": { + "type": "number" + }, + "data": { + "additionalProperties": {}, + "description": "Extracted fields matching the requested schema", + "type": "object" + }, + "error": { + "type": "string" + }, + "extractionNotes": { + "items": { + "type": "string" + }, + "type": "array" + }, + "extraction_method": { + "description": "\"llm\" | \"css_fallback\" | \"none\"", + "type": "string" + }, + "processingTime": { + "type": "number" + }, + "schema_used": { + "additionalProperties": {}, + "type": "object" + }, + "url": { + "type": "string" + }, + "validation": { + "additionalProperties": true, + "properties": { + "errors": { + "items": { + "type": "string" + }, + "type": "array" + }, + "valid": { + "type": "boolean" + } + }, + "type": "object" + } + }, + "type": "object" +}
- Changed
extract_text1 field changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
- Changed
extract_with_llm1 field changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
- Changed
fetch_url1 field changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
- Changed
generate_llms_txt4 fields changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema" - changed
Input schema / properties / analysisOptions / properties / checkSecurity / defaultPrevious value: -trueNew value: +false - added
Input schema / properties / analysisOptions / properties / probeRateLimitAdded value: +{ + "default": false, + "type": "boolean" +} - added
Input schema / properties / outputOptions / properties / robotsStyleAdded value: +{ + "default": false, + "type": "boolean" +}
- Changed
get_batch_results1 field changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
- Changed
list_ollama_models1 field changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
- Changed
localization3 fields changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema" - changed
Input schema / properties / content / descriptionPrevious value: -"Content for auto-detection of language and locale"New value: +"Page content (HTML or plain text) to analyze — required for auto_detect; no fetching is performed" - changed
Input schema / properties / url / descriptionPrevious value: -"URL for geo-blocking detection or auto-detection"New value: +"URL — required for handle_geo_blocking; for auto_detect it is optional metadata used only as a TLD country hint (the page is never fetched)"
- Changed
map_site2 fields changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema" - changed
Output schema / (root)Previous value: -nullNew value: +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "additionalProperties": false, + "properties": { + "_cost": { + "additionalProperties": true, + "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result", + "properties": { + "actual": { + "description": "Credits actually charged (0 in creator mode, half-rate on error)", + "type": "number" + }, + "projected": { + "description": "Credits projected for this call before execution", + "type": "number" + }, + "projection_note": { + "description": "Human-readable note about how the cost was projected", + "type": "string" + }, + "remaining_credits": { + "description": "Credits remaining on the account after this call, if known", + "type": [ + "number", + "null" + ] + } + }, + "type": "object" + }, + "base_url": { + "type": "string" + }, + "domain_filter_config": { + "anyOf": [ + {}, + { + "type": "null" + } + ] + }, + "filter_stats": { + "anyOf": [ + {}, + { + "type": "null" + } + ] + }, + "metadata": { + "additionalProperties": {}, + "description": "Per-URL metadata when include_metadata=true", + "type": "object" + }, + "ranked_urls": { + "description": "Present only when the `search` param was set", + "items": { + "additionalProperties": true, + "properties": { + "score": { + "type": "number" + }, + "url": { + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + }, + "site_map": { + "additionalProperties": true, + "properties": { + "depth_levels": { + "additionalProperties": {}, + "type": "object" + }, + "root": { + "items": { + "type": "string" + }, + "type": "array" + }, + "sections": { + "additionalProperties": {}, + "type": "object" + } + }, + "type": "object" + }, + "statistics": { + "additionalProperties": true, + "properties": { + "average_depth": { + "type": "number" + }, + "file_extensions": { + "additionalProperties": { + "type": "number" + }, + "type": "object" + }, + "max_depth": { + "type": "number" + }, + "query_parameters": { + "type": "number" + }, + "secure_urls": { + "type": "number" + }, + "total_urls": { + "type": "number" + }, + "unique_paths": { + "type": "number" + }, + "url_lengths": { + "additionalProperties": true, + "properties": { + "average": { + "type": "number" + }, + "max": { + "type": "number" + }, + "min": { + "type": [ + "number", + "null" + ] + } + }, + "type": "object" + } + }, + "type": "object" + }, + "total_urls": { + "type": "number" + }, + "urls": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "additionalProperties": { + "items": { + "type": "string" + }, + "type": "array" + }, + "type": "object" + } + ], + "description": "Flat array of URLs, or grouped-by-path object when group_by_path=true (default)" + } + }, + "type": "object" +}
- Changed
process_document2 fields changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema" - changed
Input schema / properties / options / descriptionPrevious value: -"Additional processing options (maxPages, pageRange:{start,end}, extractText, extractMetadata, password, outputFormat, ...)"New value: +"Additional processing options (maxPages, pageRange:{start,end}, extractText, extractMetadata, outputFormat, ...)"
- Changed
scrape2 fields changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema" - changed
Output schema / (root)Previous value: -nullNew value: +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "additionalProperties": false, + "properties": { + "_cost": { + "additionalProperties": true, + "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result", + "properties": { + "actual": { + "description": "Credits actually charged (0 in creator mode, half-rate on error)", + "type": "number" + }, + "projected": { + "description": "Credits projected for this call before execution", + "type": "number" + }, + "projection_note": { + "description": "Human-readable note about how the cost was projected", + "type": "string" + }, + "remaining_credits": { + "description": "Credits remaining on the account after this call, if known", + "type": [ + "number", + "null" + ] + } + }, + "type": "object" + }, + "content": { + "additionalProperties": true, + "description": "One key per requested format", + "properties": { + "branding": { + "additionalProperties": {}, + "description": "Static design tokens: colors, fonts, logo", + "type": "object" + }, + "html": { + "type": "string" + }, + "json": { + "description": "Result of the {type:\"json\"} format (LLM-structured extraction)" + }, + "links": { + "additionalProperties": true, + "properties": { + "external_count": { + "type": "number" + }, + "internal_count": { + "type": "number" + }, + "links": { + "items": { + "additionalProperties": true, + "properties": { + "href": { + "type": "string" + }, + "is_external": { + "type": "boolean" + }, + "original_href": { + "type": "string" + }, + "text": { + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + }, + "total_count": { + "type": "number" + } + }, + "type": "object" + }, + "markdown": { + "type": "string" + }, + "metadata": { + "additionalProperties": true, + "properties": { + "author": { + "type": "string" + }, + "canonical_url": { + "type": "string" + }, + "description": { + "type": "string" + }, + "json_ld": { + "items": {}, + "type": "array" + }, + "keywords": { + "items": { + "type": "string" + }, + "type": "array" + }, + "microdata": { + "items": {}, + "type": "array" + }, + "og_tags": { + "additionalProperties": {}, + "type": "object" + }, + "robots": { + "type": "string" + }, + "title": { + "type": "string" + }, + "twitter_tags": { + "additionalProperties": {}, + "type": "object" + }, + "url": { + "type": "string" + }, + "viewport": { + "type": "string" + } + }, + "type": "object" + }, + "rawHtml": { + "type": "string" + }, + "screenshots": { + "description": "Present for the \"screenshot\" format; each item carries a resourceUri once published", + "items": { + "additionalProperties": true, + "properties": {}, + "type": "object" + }, + "type": "array" + }, + "text": { + "type": "string" + } + }, + "type": "object" + }, + "success": { + "description": "Whether the scrape completed", + "type": "boolean" + }, + "url": { + "description": "Final URL after redirects", + "type": "string" + }, + "warnings": { + "description": "Per-format warnings; partial success never fails the whole call", + "items": { + "type": "string" + }, + "type": "array" + } + }, + "type": "object" +}
- Changed
scrape_structured1 field changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
- Changed
scrape_template1 field changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
- Changed
scrape_with_actions3 fields changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema" - added
Input schema / properties / actions / items / properties / xAdded value: +{ + "description": "scroll: absolute X coordinate to scroll to (window.scrollTo; with y, takes precedence over direction/distance)", + "minimum": 0, + "type": "number" +} - added
Input schema / properties / actions / items / properties / yAdded value: +{ + "description": "scroll: absolute Y coordinate to scroll to (window.scrollTo; with x, takes precedence over direction/distance)", + "minimum": 0, + "type": "number" +}
- Changed
search_web2 fields changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema" - changed
Output schema / (root)Previous value: -nullNew value: +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "additionalProperties": false, + "properties": { + "_cost": { + "additionalProperties": true, + "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result", + "properties": { + "actual": { + "description": "Credits actually charged (0 in creator mode, half-rate on error)", + "type": "number" + }, + "projected": { + "description": "Credits projected for this call before execution", + "type": "number" + }, + "projection_note": { + "description": "Human-readable note about how the cost was projected", + "type": "string" + }, + "remaining_credits": { + "description": "Credits remaining on the account after this call, if known", + "type": [ + "number", + "null" + ] + } + }, + "type": "object" + }, + "cached": { + "type": "boolean" + }, + "effective_query": { + "description": "Present when query expansion changed the query actually used", + "type": "string" + }, + "expanded_queries": { + "items": { + "type": "string" + }, + "type": "array" + }, + "limit": { + "type": "number" + }, + "localization": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "applied": { + "type": "boolean" + }, + "countryCode": { + "type": "string" + }, + "geoTargeting": { + "type": "boolean" + }, + "language": { + "type": "string" + }, + "searchDomain": { + "type": "string" + } + }, + "type": "object" + }, + { + "type": "null" + } + ] + }, + "offset": { + "type": "number" + }, + "processing": { + "additionalProperties": true, + "properties": { + "deduplication": { + "anyOf": [ + { + "additionalProperties": {}, + "type": "object" + }, + { + "type": "null" + } + ] + }, + "localization_applied": { + "type": "boolean" + }, + "query_expansion": { + "anyOf": [ + { + "additionalProperties": {}, + "type": "object" + }, + { + "type": "null" + } + ] + }, + "ranking": { + "anyOf": [ + { + "additionalProperties": {}, + "type": "object" + }, + { + "type": "null" + } + ] + } + }, + "type": "object" + }, + "provider": { + "additionalProperties": true, + "properties": { + "backend": { + "type": "string" + }, + "capabilities": { + "additionalProperties": {}, + "type": "object" + }, + "instanceUrl": { + "type": [ + "string", + "null" + ] + }, + "name": { + "type": "string" + }, + "note": { + "type": "string" + } + }, + "type": "object" + }, + "query": { + "type": "string" + }, + "results": { + "items": { + "additionalProperties": true, + "properties": { + "displayLink": { + "type": "string" + }, + "formattedUrl": { + "type": "string" + }, + "htmlSnippet": { + "type": "string" + }, + "link": { + "type": "string" + }, + "metadata": { + "additionalProperties": {}, + "type": "object" + }, + "pagemap": { + "additionalProperties": {}, + "type": "object" + }, + "snippet": { + "type": "string" + }, + "title": { + "type": "string" + } + }, + "type": "object" + }, + "type": "array" + }, + "search_time": { + "type": "number" + }, + "total_results": { + "type": [ + "string", + "number" + ] + } + }, + "type": "object" +}
- Changed
serp_rank2 fields changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema" - changed
Output schema / (root)Previous value: -nullNew value: +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "additionalProperties": false, + "properties": { + "_cost": { + "additionalProperties": true, + "description": "Cost-transparency metadata (D3.5), present when injected into the text copy of the result", + "properties": { + "actual": { + "description": "Credits actually charged (0 in creator mode, half-rate on error)", + "type": "number" + }, + "projected": { + "description": "Credits projected for this call before execution", + "type": "number" + }, + "projection_note": { + "description": "Human-readable note about how the cost was projected", + "type": "string" + }, + "remaining_credits": { + "description": "Credits remaining on the account after this call, if known", + "type": [ + "number", + "null" + ] + } + }, + "type": "object" + }, + "allPositions": { + "description": "Every position the target holds on this SERP", + "items": { + "additionalProperties": true, + "properties": { + "domain": { + "type": "string" + }, + "position": { + "type": [ + "number", + "null" + ] + }, + "rankAbsolute": { + "type": [ + "number", + "null" + ] + }, + "snippet": { + "type": [ + "string", + "null" + ] + }, + "title": { + "type": [ + "string", + "null" + ] + }, + "url": { + "type": [ + "string", + "null" + ] + } + }, + "type": "object" + }, + "type": "array" + }, + "checkUrl": { + "description": "Link to view the real SERP on DataForSEO", + "type": "string" + }, + "checkedAt": { + "type": "string" + }, + "configured": { + "description": "False when DATAFORSEO_LOGIN/PASSWORD are unset — no rank was fabricated", + "type": "boolean" + }, + "cost": { + "description": "USD charged by DataForSEO for this lookup (separate from CrawlForge credits)", + "type": "number" + }, + "depthScanned": { + "type": "number" + }, + "device": { + "type": "string" + }, + "found": { + "description": "Whether the target appeared anywhere in the scanned SERP", + "type": "boolean" + }, + "keyword": { + "type": "string" + }, + "location": {}, + "note": { + "description": "Present when configured=false, explains how to enable", + "type": "string" + }, + "organicResults": { + "type": "number" + }, + "position": { + "description": "Best (lowest) organic rank; null = not within top `depth`", + "type": [ + "number", + "null" + ] + }, + "rankAbsolute": { + "type": [ + "number", + "null" + ] + }, + "results": { + "description": "Top organic competitors as Google actually ranks them (capped)", + "items": { + "$ref": "#/properties/allPositions/items" + }, + "type": "array" + }, + "seResultsCount": { + "type": "number" + }, + "target": { + "description": "Bare target domain, normalized", + "type": "string" + }, + "title": { + "type": [ + "string", + "null" + ] + }, + "url": { + "description": "URL of the target's best-ranking result", + "type": [ + "string", + "null" + ] + } + }, + "type": "object" +}
- Changed
stealth_mode1 field changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
- Changed
summarize_content2 fields changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema" - changed
Input schema / properties / options / additionalPropertiesPrevious value: -falseNew value: +true
- Changed
track_changes1 field changed- changed
Input schema / $schemaPrevious value: -"http://json-schema.org/draft-07/schema#"New value: +"https://json-schema.org/draft/2020-12/schema"
27 tool updates
v4.10.0- First observed
agent - First observed
analyze_content - First observed
batch_scrape - First observed
crawl_deep - First observed
deep_research - First observed
extract_content - First observed
extract_links - First observed
extract_metadata - First observed
extract_structured - First observed
extract_text - First observed
extract_with_llm - First observed
fetch_url - First observed
generate_llms_txt - First observed
get_batch_results - First observed
list_ollama_models - First observed
localization - First observed
map_site - First observed
process_document - First observed
scrape - First observed
scrape_structured - First observed
scrape_template - First observed
scrape_with_actions - First observed
search_web - First observed
serp_rank - First observed
stealth_mode - First observed
summarize_content - First observed
track_changes
TDQS
Scored across 31 tools
The suite contains many near-overlapping read tools (scrape, extract_text, extract_content, extract_metadata, extract_links, fetch_url) plus parallel extraction paths (extract_structured, scrape_structured, extract_with_llm) and parallel autonomous paths (agent, deep_research, search_web). To their credit, the descriptions explicitly cross-reference each other with 'Not for X, use Y instead' guidance, which materially reduces misselection, but the sheer density of similar options still leaves real ambiguity. Browser_session vs scrape_with_actions vs stealth_mode also blur at the edges.
Names are uniformly lowercase snake_case, which is a clear, readable convention. Most follow a verb_noun or noun_action pattern (extract_text, fetch_url, search_web, map_site, track_changes), but a handful are bare nouns or single verbs (agent, scrape, browser_session, localization, stealth_mode, reddit_search), so the pattern is not strictly predictable. Minor deviations only.
31 tools is well above the comfortable range and forces the agent to reason over a large decision surface before every call. Several tools are thin variants that only exist because they wrap options already available inside scrape or scrape_with_actions, so the count is heavier than the domain strictly requires.
Coverage is broad and coherent for a scraping/research platform: single-page, batch, deep crawl, site mapping, JS interaction, stealth, structured/LLM extraction, templates, documents, change tracking, research, Reddit, SERP rank, localization, and result retrieval. Gaps are minor - no explicit job listing/cancellation for async batches, and no usage/credit introspection - but these are workaroundable.
Maintenance
Related MCP Connectors
Web MCP: scrape/crawl sites, web search, brand assets, app stores, YouTube, Reddit, Hacker News.
5113Scrape, crawl and search the web for AI agents via MCP.
Web data tools for AI agents: pages as markdown, search, maps, commerce, jobs, AI answers.
Web search, URL content extraction to Markdown, site mapping, and recursive web crawler.
Related MCP Servers
- FlicenseNot gradedqualityCmaintenanceProvides 42+ MCP tools for browser automation, web scraping, and search, enabling AI agents like Claude and Cursor to browse, extract data, and run research agents on the live web.8-
- AlicenseBqualityAmaintenanceEnables AI assistants to crawl websites, extract dynamic content, navigate links, and save structured Markdown files via the MCP protocol, with support for anti-bot bypass, CSS selectors, and custom JavaScript execution.142MIT

Scout MCP Serverofficial
AlicenseNot gradedqualityCmaintenanceEnables web search, scraping, extraction, and crawling through an MCP interface, allowing coding agents to access real-time web data.1MIT- AlicenseNot gradedqualityBmaintenanceA comprehensive web scraping MCP server with 26 tools for fetching, parsing, extracting, and assisting with web content, returning Markdown-formatted results.MIT