Crawl-MCP
Crawl-MCP is an MCP server that crawls, searches, extracts, summarizes, and persists web/PDF/Office/YouTube content with token-saving options.
Crawl web pages with JavaScript rendering, CSS selectors, screenshots, media extraction, bot-detection fallback, and recursive site crawls.
Search the web through Google Custom Search, Brave, Exa, or SearXNG, with genre filters, fallback engines, and options to crawl top results.
Extract LLM-guided data, entities such as emails/phones/prices, CSS/LLM/table structured data, and large-content chunks with BM25 filtering.
Process PDFs, Word, Excel, PowerPoint, and ZIP archives into markdown.
Extract YouTube transcripts, video metadata, comments, batch transcripts, and optional LLM summaries/translations.
Run batch crawls, batch searches, multi-URL pattern-based crawls, and batch YouTube transcript extraction.
Persist full unsliced results to disk with
output_pathwhile returning slim metadata to reduce context/token usage.Optionally auto-summarize large content with an LLM and return structured error dictionaries on failures.
Enables Google Search integration with 7 optimized search genres using official Google operators, including tools for single searches, batch searches, and combined search-and-crawl operations with metadata extraction.
Extracts video transcripts and summaries from YouTube videos without requiring API keys, supporting multi-language transcript extraction and batch processing of multiple videos.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Crawl-MCPsummarize this article about AI advancements from https://example.com/ai-news"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Crawl-MCP: Unofficial MCP Server for crawl4ai
This is an unofficial MCP server for the crawl4ai library. It is not affiliated with the crawl4ai project.
A Model Context Protocol (MCP) server that wraps crawl4ai. It extracts content from web pages, PDFs, Office documents, YouTube videos, and web search results (Google Custom Search, Brave, Exa, or a self-hosted SearXNG). To keep token usage low, it can summarize content with an LLM (auto_summarize, opt-in) and write full results to disk (output_path).
Key Features
Web search through Google Custom Search, Brave, Exa, or a self-hosted SearXNG instance, with automatic fallback between engines and 6 genres (file type and language filters)
Web crawling with JavaScript rendering, recursive site crawls, and entity extraction
Content extraction from web pages, PDFs, Word, Excel, PowerPoint, and ZIP archives
LLM summarization when
auto_summarize=true. Without it, large responses are truncated at about 25000 tokens, andoutput_pathsaves the full contentDisk persistence: full results go to a file and the response carries only metadata, so an agent reads the file when it needs the content
YouTube transcripts and summaries without an API key
21 tools that return structured error dicts on failure
Related MCP server: Crawl4AI MCP Server
Quick Start
Prerequisites (Required First)
Python 3.11 or later (FastMCP requires Python 3.11+)
Install the system dependencies for Playwright.
Ubuntu 24.04 LTS needs a manual setup:
# Manual setup required due to t64 library transition
sudo apt update && sudo apt install -y \
libnss3 libatk-bridge2.0-0 libxss1 libasound2t64 \
libgbm1 libgtk-3-0t64 libxshmfence-dev libxrandr2 \
libxcomposite1 libxcursor1 libxdamage1 libxi6 \
fonts-noto-color-emoji fonts-unifont python3-venv python3-pip
python3 -m venv venv && source venv/bin/activate
pip install playwright==1.64.1 && playwright install chromium
sudo playwright install-depsOther Linux distributions (installs Chromium and its system libraries):
uvx --from playwright==1.64.1 playwright install --with-deps chromiummacOS / Windows:
uvx --from playwright==1.64.1 playwright install chromiumIf Playwright is already installed in a Python environment, python -m playwright install --with-deps chromium (Linux) or playwright install chromium (macOS/Windows) works as well.
Installation
UVX (recommended):
# After system preparation above - that's it!
uvx --from git+https://github.com/walksoda/crawl-mcp crawl-mcpDocker:
# Clone the repository
git clone https://github.com/walksoda/crawl-mcp
cd crawl-mcp
# Build and run with Docker Compose (STDIO mode)
docker-compose up --build
# Or build and run HTTP mode on port 8000
docker-compose --profile http up --build crawl4ai-mcp-http
# Or build manually
docker build -t crawl4ai-mcp .
docker run -it crawl4ai-mcpThe Docker image includes headless Chromium, Firefox, and WebKit, plus Google Chrome Stable. Browser flags are preset for running in a container, the server runs as a non-root user, and all required system libraries are installed.
Claude Desktop Setup
For a UVX installation, add this to your claude_desktop_config.json:
{
"mcpServers": {
"crawl-mcp": {
"transport": "stdio",
"command": "uvx",
"args": [
"--from",
"git+https://github.com/walksoda/crawl-mcp",
"crawl-mcp"
]
}
}
}For Docker in HTTP mode:
{
"mcpServers": {
"crawl-mcp": {
"transport": "http",
"baseUrl": "http://localhost:8000"
}
}
}Documentation
Topic | Description |
Installation on each platform | |
Parameters and behavior of every tool | |
Client and platform configurations | |
Running the server over HTTP and calling it | |
Workflows that combine tools | |
Development setup and contributing |
Language-Specific Documentation
Tool Overview
Web Crawling (3)
crawl_url- Extract web page content with JavaScript support (remote.html/.htmURLs are rendered in the browser; document URLs are converted as files)deep_crawl_site- Crawl multiple pages from a site with configurable depth (max_depth1-2,max_pages1-10)crawl_url_with_fallback- Crawl with fallback strategies for anti-bot sites
Data Extraction (3)
intelligent_extract- Extract specific data from web pages using LLMextract_entities- Extract entities (emails, phones, etc.) from web pagesextract_structured_data- Extract structured data using CSS selectors or LLM
YouTube (4)
extract_youtube_transcript- Extract YouTube transcripts with timestamps and timezone-converted publish time (timezone)batch_extract_youtube_transcripts- Extract transcripts from multiple YouTube videos (max 3)get_youtube_video_info- Get YouTube video metadata (includingpublished_atin the requestedtimezone) and transcript availabilityextract_youtube_comments- Extract YouTube video comments with pagination
Search (6)
search_web- Web search with optional genre filteringbatch_search_web- Run multiple web searches (max 3)search_and_crawl- Web search and crawl of the top resultsget_search_genres- Get available search genres and the engine configurationsearch_google,batch_search_google- Deprecated aliases ofsearch_webandbatch_search_web
The search tools need at least one configured engine: GOOGLE_SEARCH_API_KEY and GOOGLE_SEARCH_ENGINE_ID (Google Custom Search, closed to new customers and ending on 2027-01-01), BRAVE_SEARCH_API_KEY, EXA_API_KEY, or SEARXNG_URL (a self-hosted SearXNG instance, no API key). SEARCH_ENGINE sets the default engine (auto tries the configured engines in SEARCH_ENGINE_ORDER, default google_custom,brave,exa,searxng), and the engine request key overrides it for one call. See the API Reference.
search_web, batch_search_web and search_and_crawl accept the same search_genre values: pdf, documents, presentations, spreadsheets, japanese, english. Matching is case-insensitive. Any other value returns an invalid_search_genre error before the search runs. Exa cannot filter by file type, so auto skips Exa for the file type genres.
result_type adds Brave result sections to a search: discussions (forum posts with forum name, answer count and score), faq (question and answer excerpts), videos and news. Several can be requested at once, for example ["web", "discussions", "faq"]; the extra sections come back in extra_results. They need the brave engine. discussions, faq and videos share one Brave request and appear for some queries only; news is a separate request to Brave News Search.
File Processing (3)
process_file- Convert PDF, Word, Excel, PowerPoint, ZIP to markdownget_supported_file_formats- Get supported file formats and capabilitiesenhanced_process_large_content- Process large content with chunking and BM25 filtering
Batch Operations (2)
batch_crawl- Crawl multiple URLs sequentially with fallback (max 3 URLs)multi_url_crawl- Multi-URL crawl with pattern-based config (max 5 URL patterns, processed sequentially)
Persist Large Results to Disk
All information-gathering tools accept an optional output_path parameter. The tool writes the full content to disk and returns only metadata. An LLM can then fetch large pages, long YouTube transcripts, or whole batches without filling its context, and read the saved file when it needs the content.
How it works:
Single-file tools (e.g.
crawl_url,extract_youtube_transcript) write one.md(or.jsonfor JSON-kind tools). Pass an absolute file path; the extension is added if omitted. An existing regular file at that path is rejected unlessoverwrite=true.Batch tools (
batch_crawl,multi_url_crawl,deep_crawl_site,search_and_crawl,batch_extract_youtube_transcripts) expect an absolute directory path and write one.mdper URL plusindex.json. Any non-existent path is treated as a directory and created, including names containing dots such as/tmp/run.v1. If the path already exists as a regular file, the call is rejected.batch_crawl/multi_url_crawlkeep theirlistreturn shape and embed anoutput_filekey on each success item.Request-dict tools (
search_google,batch_search_google,search_and_crawl,batch_extract_youtube_transcripts) read the persistence keys directly from their request dict.Common parameters:
output_path(absolute; defaultNone,""also skips persistence),include_content_in_response(defaultfalse; whentrue, the content is also included in the response, still subject to anycontent_limit/content_offset/max_content_per_pageslicing),overwrite(defaultfalse).Writes are atomic per file (temp file +
os.replace); parent directories are auto-created; the full unsliced payload is written before any slicing or tool-internal truncation, so the file is complete even when the response is sliced.Batch dict tools (
deep_crawl_site,search_and_crawl,batch_extract_youtube_transcripts) skip per-item persistence for items that reportsuccess=false; these still appear inindex.jsonwithfile: null, so callers can see every attempted item. (batch_crawl/multi_url_crawlalso skip failed items and record them inindex.jsonwithoutput_file: null.)
Single Markdown file:
{
"tool": "crawl_url",
"arguments": {
"url": "https://example.com/long-article",
"output_path": "/tmp/crawl_out/article.md"
}
}JSON structured extraction (the extension is added automatically):
{
"tool": "extract_structured_data",
"arguments": {
"url": "https://example.com/products",
"extraction_type": "css",
"css_selectors": {"price": ".price", "name": "h1"},
"output_path": "/tmp/crawl_out/products"
}
}Batch directory mode:
{
"tool": "batch_crawl",
"arguments": {
"urls": ["https://a.example", "https://b.example"],
"output_path": "/tmp/crawl_out/batch_run1"
}
}Each persisted markdown file begins with a YAML frontmatter block containing url, title, fetched_at, and source_tool.
Common Use Cases
Content research:
search_and_crawl → extract_structured_data → analysisCollecting documentation:
deep_crawl_site → batch processing → extractionVideo analysis:
extract_youtube_transcript → summarization workflowSite mapping:
batch_crawl → multi_url_crawl → aggregated dataQuick Troubleshooting
Installation problems:
Re-run the Playwright install commands above with proper privileges
Try the development installation method
Check that the browser dependencies are installed
Slow or incomplete pages:
Use
wait_for_js: truefor JavaScript-heavy sitesIncrease timeout for slow-loading pages
Use
extract_structured_datafor targeted extraction
Configuration problems:
Check JSON syntax in
claude_desktop_config.jsonVerify file paths are absolute
Restart Claude Desktop after configuration changes
Project Structure
The underlying library is crawl4ai by unclecode. This repository (walksoda) is an unofficial third-party MCP wrapper around it.
License
This project is an unofficial wrapper around the crawl4ai library. See the crawl4ai license for the underlying functionality.
Contributing
See the Development Guide for development setup and contribution guidelines.
Related Projects
crawl4ai - The underlying web crawling library
Model Context Protocol - The standard this server implements
Claude Desktop - Primary client for MCP servers
Available Tools
21 toolsbatch_crawlA
Crawl multiple URLs with fallback. Max 3 URLs per call. Use output_path (directory) to persist full per-URL markdown + index.json; the return shape stays a list, each success item gets an output_file key.
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | URLs to crawl (max 3) | |
| overwrite | No | Overwrite existing per-URL files inside output_path. Defaults to False (existing files cause an output_path_exists error, returned as a single-element list). | |
| output_path | No | Absolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names like /tmp/run.v1 are fine). The list return shape is preserved; each successful item gains an 'output_file' key. Failed items (success=False) are NOT written as .md but still appear in index.json with output_file=null. | |
| wait_for_js | No | Wait for JS (default: False) | |
| base_timeout | No | Timeout per URL (default: 30) | |
| extract_media | No | Extract media (default: False) | |
| generate_markdown | No | Generate markdown (default: True) | |
| include_content_in_response | No | When True (with output_path), keep full markdown/content in each list item. Defaults to False so the response stays token-efficient. |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does meaningful work: it discloses the fallback behavior, the 3-URL cap, that return shape stays a list, and that successes gain an output_file key. It omits failure/partial-failure semantics and any auth or rate-limit context, which keeps it below 5.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences, front-loaded with the core action and limit, then the persistence/return-shape contract. No filler and no restatement of the tool name.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists and the 8 parameters are fully documented, so return values need not be re-explained; the description covers the limits and the persistence contract well. What's missing is routing guidance against the many sibling crawl/search tools.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the schema descriptions are unusually thorough (overwrite error semantics, output_path rejection rules, include_content_in_response token tradeoff). The description restates output_path persistence and adds the output_file key detail, but adds little beyond what the schema already documents.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Crawl multiple URLs') plus a distinguishing trait ('with fallback') and a hard limit ('Max 3 URLs per call'). It does not name the nearest siblings (multi_url_crawl, batch_search_web, deep_crawl_site), so the agent must still infer which batch tool to pick.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives an actionable usage cue for output_path ('Use output_path (directory) to persist...') and a size constraint, which implies the batch-of-few use case. But there is no explicit when-to-use/when-not versus multi_url_crawl or deep_crawl_site, and no stated prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
batch_extract_youtube_transcriptsARead-only
Extract transcripts from multiple YouTube videos. Max 3 URLs per call. Supply output_path (directory) in the request to persist per-video markdown files + index.json and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| request | Yes | Dict with: urls (max 3), languages, include_timestamps, timezone (IANA name e.g. 'Asia/Tokyo' for converting each video's publish time under metadata.published_at; defaults to 'UTC'). Optional persistence keys: output_path (absolute directory — per-video .md files + index.json; dot-containing dir names are fine), include_content_in_response (bool; default False — when True, per-video transcripts stay in the response as well), overwrite (bool; default False — existing files rejected). Failed items (success=False) are recorded in index.json with file=null but no .md is written. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the readOnlyHint annotation, the description discloses that passing output_path causes per-video .md files plus index.json to be written and changes the response shape — behavior an agent cannot see from the schema annotations. The schema adds overwrite=false rejection and failed-item recording, and the description summarizes the persistence behavior well. Note the mild tension: filesystem writes are disclosed while readOnlyHint=true, but since persistence is strictly opt-in and the default call is a pure read, this is disclosed rather than contradicted.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, zero filler, with the hard constraint (max 3 URLs) and the highest-leverage optional behavior (output_path) placed early. Every sentence carries information the agent needs to act.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be explained, and the description covers the batching limit and the persistence side effect, which are the two things that most change how the tool is called. What remains missing is guidance on handling more than 3 URLs and how partial failures surface to the caller, a minor gap given the rich schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the request object's description already enumerates urls, languages, include_timestamps, timezone, output_path, include_content_in_response, and overwrite. The description only re-states the URL cap and output_path behavior, adding no format or edge-case detail beyond the schema; baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource ("Extract transcripts from multiple YouTube videos"), and the word "multiple" plus the batch_ name prefix distinguishes it in practice from the single-video sibling extract_youtube_transcript. It never names that sibling explicitly, so an agent must infer the split, keeping this short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives an operational constraint ("Max 3 URLs per call") and explains the output_path option's effect (persist files, get a slim response), which is real usage guidance. However, it never says when to prefer this over extract_youtube_transcript or how to handle larger batches (chunking? repeat calls?), leaving the key routing and batching decisions to inference.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
batch_search_googleARead-only
Deprecated alias of batch_search_web (same request keys and results). Use batch_search_web.
| Name | Required | Description | Default |
|---|---|---|---|
| request | Yes | Dict with: queries (max 3), num_results_per_query, search_genre (optional, one of: pdf, documents, presentations, spreadsheets, japanese, english; adds a filetype/language operator to the query, other values return an error), engine (optional, one of: auto, google_custom, brave, exa, searxng; default from SEARCH_ENGINE, auto tries the configured engines in SEARCH_ENGINE_ORDER and moves to the next one on errors other than zero results), result_type (optional; default web; one of web, discussions, faq, videos, news, or several as a list or a comma-separated string; everything except web runs on brave only; news is a separate Brave News Search request; results holds web when requested, else the first type, and the other types go to extra_results; discussions are forum posts with the forum name, answer count, post score, question and top comment; faq are question and answer excerpts; Brave returns these sections for some queries only, and num_results only shortens them), recent_days. Optional persistence keys: output_path (absolute file path, auto .json extension — full result set written to disk), include_content_in_response (bool, default False — when True keeps results in the response too), overwrite (bool, default False). |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The deprecation notice and the equivalence claim ('same request keys and results') are genuine behavioral facts an agent could not infer from annotations or schema. With readOnlyHint=true already covering the safety profile, this is strong, though it does not state whether the alias may be removed or behave differently over time.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two short sentences, zero waste, with the deprecation status front-loaded before the redirect instruction.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a deprecated alias stub, the description supplies everything needed: the equivalence guarantee, the redirection target, and an output schema to explain results. Nothing an agent requires to decide or call correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the nested request object is exhaustively documented, so the schema carries parameter meaning. The description only asserts the keys match batch_search_web, adding no new syntax or format detail beyond that pointer.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States precisely what the tool is (a deprecated alias of batch_search_web) and that it takes the same request keys and returns the same results, so an agent immediately knows the underlying resource. It also names the sibling it duplicates, making it unambiguous against the other search tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The final clause, 'Use batch_search_web,' is an explicit directive naming the preferred alternative. There is no ambiguity about when to select this tool versus its replacement.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
batch_search_webARead-only
Perform multiple web searches. Max 3 queries per call. Supply output_path in the request to persist the full result set to disk as JSON and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| request | Yes | Dict with: queries (max 3), num_results_per_query, search_genre (optional, one of: pdf, documents, presentations, spreadsheets, japanese, english; adds a filetype/language operator to the query, other values return an error), engine (optional, one of: auto, google_custom, brave, exa, searxng; default from SEARCH_ENGINE, auto tries the configured engines in SEARCH_ENGINE_ORDER and moves to the next one on errors other than zero results), result_type (optional; default web; one of web, discussions, faq, videos, news, or several as a list or a comma-separated string; everything except web runs on brave only; news is a separate Brave News Search request; results holds web when requested, else the first type, and the other types go to extra_results; discussions are forum posts with the forum name, answer count, post score, question and top comment; faq are question and answer excerpts; Brave returns these sections for some queries only, and num_results only shortens them), recent_days. Optional persistence keys: output_path (absolute file path, auto .json extension — full result set written to disk), include_content_in_response (bool, default False — when True keeps results in the response too), overwrite (bool, default False). |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
readOnlyHint=true is consistent with a search operation, so no contradiction. The description contributes real behavior beyond the annotations: the 3-query cap and the fact that supplying output_path makes the response 'slim' while the full set lands on disk - a consequence the annotation does not convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, zero waste, and the primary action is front-loaded ahead of the operational constraints. Nothing is buried or repeated from the schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given a rich nested schema at 100% coverage and an output schema, the description needs only the operational gotchas - and it supplies the two most important (query cap, persistence behavior). It could still note that result_type alternatives run on Brave only, but that detail lives in the schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% and the nested request object documents every key, so baseline would be 3. The description earns extra credit for explaining the observable effect of output_path (slim response), which the schema states only as 'full result set written to disk'.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Perform multiple web searches') plus a hard scope limit ('Max 3 queries per call'). It does not explicitly name siblings like search_web or batch_search_google, so the batch-vs-single distinction is left to the name rather than the description.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The only concrete guidance is a usage rule for persistence ('Supply output_path ... to persist the full result set'), which covers one scenario well. It never says when to choose this over search_web or batch_search_google, so the selection decision must be inferred from the name.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_urlA
Extract web page content with JavaScript support. Use wait_for_js=true for SPAs. Use content_offset/content_limit to paginate the response. Use output_path to persist the full unsliced content to disk as markdown and receive a slim metadata-only response.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to crawl | |
| timeout | No | Timeout in seconds | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False (existing files are rejected before any fetch). | |
| output_path | No | Absolute file path (auto .md extension) to persist the full unsliced markdown. When set, the response is slimmed to metadata+file path to save tokens. content_limit/content_offset still affect the response copy but not the on-disk file. | |
| wait_for_js | No | Wait for JavaScript | |
| css_selector | No | CSS selector for extraction | |
| content_limit | No | Max characters to return (0=unlimited) | |
| extract_media | No | Extract images/videos | |
| auto_summarize | No | Auto-summarize large content | |
| content_offset | No | Start position for content (0-indexed) | |
| take_screenshot | No | Take screenshot | |
| generate_markdown | No | Generate markdown | |
| wait_for_selector | No | Wait for element to load | |
| include_cleaned_html | No | Include cleaned HTML | |
| use_undetected_browser | No | Bypass bot detection | |
| include_content_in_response | No | When True (with output_path set), keep markdown/content in the response too. Note: the response copy is still subject to content_limit/content_offset slicing; only the on-disk file holds the full unsliced payload. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description adds significant behavioral detail: how output_path slimdown works, the interaction between content_offset/limit and the on-disk file, and the include_content_in_response flag. This goes beyond basic functionality.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise: three short sentences that front-load the purpose and quickly cover the most important usage scenarios. No unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 16 parameters and an output schema, the description covers the most important behaviors (JS, pagination, persistence) but omits details on less common params like css_selector or extract_media. Still, the core agent needs are addressed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description adds value by explaining the interplay between output_path, content_offset, and content_limit, and advises when to use wait_for_js, which is not present in the schema alone.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool extracts web page content with JavaScript support, and provides specific use cases like SPAs and pagination. However, it does not differentiate from sibling tools like 'crawl_url_with_fallback' or 'deep_crawl_site', which lowers the score slightly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit usage guidelines for key features (wait_for_js for SPAs, pagination with offsets, persisting content to disk). It lacks guidance on when not to use this tool or alternatives, but provides enough context for common scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_url_with_fallbackARead-only
Crawl with fallback strategies for anti-bot sites. Use content_offset/content_limit to paginate the response. Use output_path to persist the full unsliced content to disk as markdown and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to crawl | |
| timeout | No | Timeout in seconds | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False (existing files rejected before any fetch). | |
| output_path | No | Absolute file path (auto .md extension) to persist the full unsliced markdown. When set, the response is slimmed to metadata+file path. content_limit/content_offset still affect the response copy but not the on-disk file. | |
| wait_for_js | No | Wait for JavaScript | |
| css_selector | No | CSS selector | |
| content_limit | No | Max characters to return (0=unlimited) | |
| extract_media | No | Extract media | |
| auto_summarize | No | Auto-summarize content | |
| content_offset | No | Start position for content (0-indexed) | |
| take_screenshot | No | Take screenshot | |
| generate_markdown | No | Generate markdown | |
| wait_for_selector | No | Element to wait for | |
| include_content_in_response | No | When True (with output_path set), keep markdown/content in the response too. Note: the response copy is still subject to content_limit/content_offset slicing; only the on-disk file holds the full unsliced payload. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate readOnlyHint=true, which is consistent with crawling. Description adds that fallback strategies exist for anti-bot sites, but does not detail what those strategies are or any other behavioral traits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with core purpose, no fluff. Each sentence adds distinct value: purpose, pagination, and persistence.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 14 parameters and an output schema (not shown), the description covers the main behavioral aspects (fallback, pagination, persistence). Minor gap: no explanation of the fallback strategies themselves, but overall adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 100% schema coverage, baseline is 3. The description adds meaningful context for pagination (content_offset/content_limit) and persistence (output_path with behavior rules), going beyond the schema comments.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action ('Crawl') and target ('anti-bot sites') with fallback strategies, but does not differentiate from the sibling tool 'crawl_url' which likely has similar purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage via mention of pagination and persistence, but does not provide explicit guidance on when to choose this tool over alternatives like 'crawl_url' or 'search_and_crawl'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
deep_crawl_siteARead-only
Crawl multiple pages from a site with configurable depth. Use output_path (directory) to persist per-URL markdown files + index.json; the response is then slimmed to metadata only.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Starting URL | |
| max_depth | No | Link depth, 1-2 (default: 2). Values outside the range return an error. | |
| max_pages | No | Max pages, 1-10 (default: 5). Values outside the range return an error. | |
| overwrite | No | Overwrite existing per-URL files inside output_path. Defaults to False (existing files cause an output_path_exists error). | |
| output_path | No | Absolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names like /tmp/run.v1 are fine). When set, the response is slimmed to metadata+file paths. Failed items (success=False) are NOT written as .md but still recorded in index.json with file=null. | |
| url_pattern | No | URL filter pattern | |
| base_timeout | No | Timeout per page | |
| extract_media | No | Extract media | |
| crawl_strategy | No | 'bfs'|'dfs'|'best_first' | bfs |
| score_threshold | No | Min relevance 0-1 | |
| include_external | No | Follow external links | |
| include_content_in_response | No | When True (with output_path set), also include per-page content/markdown in the response items. Defaults to False so the response stays token-efficient. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description adds real behavior the annotations don't convey: setting output_path persists per-URL markdown plus index.json, and the response is then slimmed to metadata only. That is exactly the side effect an agent needs to know before opting into file output. The only gap is that it doesn't reconcile this file-writing behavior with the readOnlyHint annotation, which reads as local-artifact generation rather than a remote-state mutation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, zero padding, with the core action front-loaded and the output_path persistence caveat immediately after. Every clause carries information an agent would otherwise have to infer.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Return values need no explanation because an output schema exists, and the schema fully covers the 12 parameters. However, for a 12-parameter tool sitting in a crowded crawl family, the description leaves the sibling-routing decision (deep_crawl_site vs batch_crawl vs multi_url_crawl vs crawl_url) entirely to the agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so the schema already documents all 12 parameters including ranges and defaults. The description nonetheless reinforces the highest-impact one — output_path as a directory with a response-shape consequence — adding value beyond the schema's own wording. Baseline 3 is exceeded slightly for that targeted emphasis.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource ('Crawl multiple pages from a site with configurable depth'), and 'multiple pages' implicitly contrasts with the single-page crawl_url sibling. It does not explicitly differentiate itself from batch_crawl, multi_url_crawl, or search_and_crawl, so an agent still has to disambiguate among the crawl family by name alone.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is only implied: crawl a site (not a single URL) to a given depth. There is no explicit 'use this when you need X instead of batch_crawl/multi_url_crawl' guidance, and no statement of prerequisites or when the tool is a poor fit (e.g., across many domains).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
enhanced_process_large_contentBRead-only
Process large content with chunking and BM25 filtering. Use output_path to persist chunks + summaries to disk as JSON and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to process | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False. | |
| output_path | No | Absolute file path (auto .json extension) to persist the full chunks + summaries as JSON. When set, the response is slimmed to metadata+file path (chunks, chunk_summaries, merged_summary, final_summary removed). | |
| filter_query | No | Keywords for BM25 filtering | |
| chunk_overlap | No | Overlap tokens | |
| merge_strategy | No | 'hierarchical'|'linear' | linear |
| max_chunk_tokens | No | Max tokens per chunk | |
| summarize_chunks | No | Summarize chunks | |
| chunking_strategy | No | 'topic'|'sentence'|'overlap'|'regex' | sentence |
| extract_top_chunks | No | Top chunks to extract | |
| filtering_strategy | No | 'bm25'|'pruning'|'llm' | bm25 |
| final_summary_length | No | 'short'|'medium'|'long' | short |
| similarity_threshold | No | Min similarity 0-1 | |
| include_content_in_response | No | When True (with output_path set), also include chunks/summaries in the response. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations declare readOnlyHint=true, but the description states the tool persists data to disk, which is a write operation. This is a direct contradiction. Additionally, the description does not disclose other behavioral traits such as required permissions, rate limits, or side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences long, directly stating the core functionality and key usage advice. Every word earns its place with no fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's 14 parameters and existence of an output schema, the description is minimal but covers the main point. However, it lacks details about return structure, error conditions, and behavioral nuances beyond the output_path feature.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Input schema has 100% coverage for all 14 parameters, providing clear descriptions. The description adds value by explaining the effect of output_path and the slim response, which enhances understanding beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it processes large content with chunking and BM25 filtering, and mentions persisting to disk. It distinguishes itself from siblings by specifying the advanced processing features, though it doesn't explicitly differentiate from similar tools like 'process_file'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. It only implies usage via the description but lacks explicit context, exclusions, or comparisons with sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_entitiesBRead-only
Extract entities (emails, phones, etc.) from web pages. Use output_path to persist the full entity extraction output to disk as JSON and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Target URL | |
| use_llm | No | Use LLM for NER | |
| llm_model | No | LLM model | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False. | |
| deduplicate | No | Remove duplicates | |
| output_path | No | Absolute file path (auto .json extension) to persist the full entity extraction as JSON. When set, the response is slimmed (content, markdown, extracted_data.raw_content removed). | |
| entity_types | Yes | Regex types: emails, phones, urls, dates, ips, prices, credit_cards, coordinates (singular forms like email/phone are accepted), plus any custom_patterns keys. Without use_llm, other types are listed under unsupported_entity_types with a warning. With use_llm=True any type name works (e.g. people, organizations, locations, products, events). | |
| llm_provider | No | LLM provider | |
| custom_patterns | No | Custom regex patterns as {type_name: regex}; list the type_name in entity_types to use it (regex mode only) | |
| include_context | No | Include context | |
| include_content_in_response | No | When True (with output_path set), also keep the entity data in the response. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only declare readOnlyHint=true, and the description adds real behavioral context: with output_path set, the full extraction is written to disk as JSON and the response is slimmed. However, this creates an unaddressed tension with readOnlyHint (a file is written) and the description says nothing about regex-vs-LLM mode consequences, which drive most of the tool's actual behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with the core action and followed by the highest-value operational hint. No filler or repetition, though the second sentence could be tighter.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For an 11-parameter tool, the description leans heavily on a fully documented schema and an existing output schema, so return values and per-parameter semantics are covered elsewhere. The remaining gap is the absence of any routing guidance among the many extraction siblings.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so every parameter including entity_types, custom_patterns, and overwrite is already documented in detail. The description's note about output_path duplicates what the schema already states, adding no syntax or default information beyond it.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Names a specific verb and resource ("Extract entities") and scopes it to web pages, with examples of the entity kinds (emails, phones). It does not explicitly differentiate itself from close siblings like extract_structured_data or intelligent_extract, which an agent could plausibly confuse it with.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no statement of when to choose this over extract_structured_data, intelligent_extract, or crawl_url, and no exclusions or prerequisites. The only guidance is operational (use output_path to get a slim response), not selection guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_structured_dataBRead-only
Extract structured data using CSS selectors or LLM. Use output_path to persist the full extraction (including table_data) to disk as JSON and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Target URL | |
| timeout | No | Timeout in seconds | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False. | |
| output_path | No | Absolute file path (auto .json extension) to persist the full extracted_data + table_data as JSON. When set, the response is slimmed (content, markdown, table_data, extracted_data.raw_content removed). | |
| wait_for_js | No | Wait for JavaScript | |
| css_selectors | No | CSS selector mapping | |
| extraction_type | No | 'css'|'llm'|'table' | css |
| extraction_schema | No | Schema definition | |
| generate_markdown | No | Generate markdown | |
| table_chunking_strategy | No | 'intelligent'|'fixed'|'semantic' | intelligent |
| use_llm_table_extraction | No | Use LLM table extraction | |
| include_content_in_response | No | When True (with output_path set), also keep extracted_data/table_data/content in the response. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations indicate readOnlyHint=true, which matches the extraction purpose. The description adds useful behavioral context about output_path persisting full data and slimming the response, but does not disclose other behaviors like rate limits or auth needs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise with two sentences, no wasted words. It front-loads the main purpose, but could be better structured by separating the core function from the output_path detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (12 parameters) and presence of an output schema, the description covers the key behavior of output_path but lacks context on when to use different extraction types or how they differ. This is adequate but not comprehensive.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents parameters thoroughly. The description adds minimal value beyond restating extraction types and output_path behavior, meeting the baseline expectation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states the tool extracts structured data using CSS selectors or LLM, which is specific and clear. It distinguishes from sibling crawling tools but does not explicitly differentiate from other extraction tools like intelligent_extract or extract_entities.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. It mentions the output_path behavior but lacks prerequisites or when-not-to-use instructions, which is a gap given the large set of sibling extraction tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_youtube_commentsBRead-only
Extract YouTube video comments. Supports pagination via comment_offset. Use output_path to persist the full unsliced comment list to disk as JSON; the response is then slimmed to metadata only.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | YouTube video URL | |
| sort_by | No | 'popular'|'recent' | popular |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False. | |
| output_path | No | Absolute file path (auto .json extension) to persist the full unsliced/untruncated comment list. When set, the complete comment array is written to disk BEFORE the internal token-limit reduction, and the response is slimmed (extracted_data.comments removed but comment_count etc. kept). | |
| max_comments | No | Max comments to retrieve (1-1000) | |
| content_limit | No | Max characters to return (0=unlimited) | |
| comment_offset | No | Number of comments to skip (for pagination) | |
| content_offset | No | Start position for content (0-indexed) | |
| include_replies | No | Include reply comments | |
| include_content_in_response | No | When True (with output_path set), keep the comment list in the response too. Note: the response copy is still subject to content_limit/content_offset slicing and the token-limit comment-array reduction; only the on-disk file holds the full list. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Description describes a side effect (writing comments to disk) that contradicts the readOnlyHint annotation, which implies no mutations. This is a serious inconsistency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences; first states core function, second explains key persistence feature. No superfluous information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers main behavior, pagination, and file persistence, but the annotation contradiction undermines completeness. Output schema exists to cover return values.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Input schema has 100% description coverage, so baseline 3. Description adds marginal value beyond schema, clarifying output_path behavior but not essential.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states verb 'Extract' and resource 'YouTube video comments'. Differentiates from sibling tools like extract_youtube_transcript.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implies usage for extracting comments with pagination and optional file persistence, but lacks explicit when-to-use or alternative comparisons.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_youtube_transcriptARead-only
Extract YouTube transcripts with timestamps. Works with public captioned videos. Supports fallback to page crawl. Use output_path to persist the full unsliced transcript to disk as markdown.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | YouTube video URL | |
| timezone | No | IANA timezone name (e.g. 'Asia/Tokyo', 'America/New_York') used to convert the video's publish time, returned under metadata.published_at (requires enrich_metadata=True). When unspecified, defaults to 'UTC' (the original YouTube offset is always kept in metadata.published_at.original). | UTC |
| languages | No | Language codes in preference order | |
| llm_model | No | LLM model | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False. | |
| output_path | No | Absolute file path (auto .md extension) to persist the full unsliced transcript. When set, the response is slimmed to metadata+file path. content_limit/content_offset still affect the response copy but not the on-disk file. | |
| llm_provider | No | LLM provider | |
| translate_to | No | Target language for translation | |
| content_limit | No | Max characters to return (0=unlimited) | |
| auto_summarize | No | Auto-summarize large content | |
| content_offset | No | Start position for content (0-indexed) | |
| summary_length | No | 'short'|'medium'|'long' | medium |
| enrich_metadata | No | Enrich metadata (upload_date, view_count) via page crawl | |
| fallback_timeout | No | Fallback crawl timeout in seconds | |
| include_metadata | No | Include video metadata | |
| include_timestamps | No | Include timestamps | |
| max_content_tokens | No | Max tokens before summarization | |
| preserve_formatting | No | Preserve formatting | |
| enable_crawl_fallback | No | Enable page crawl fallback when API fails | |
| include_content_in_response | No | When True (with output_path set), keep the transcript in the response too. Note: the response copy is still subject to content_limit/content_offset slicing; only the on-disk file holds the full transcript. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
readOnlyHint=true already tells the agent this is a safe network read. The description adds useful context – crawl fallback and that output_path writes the full unsliced transcript to disk, plus that the response copy is slimmed. However, it doesn't surface the heavier behaviors (auto_summarize, llm translate) that the tool also exposes, leaving a gap relative to what the tool can do.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four short sentences, front-loaded with the core action, no filler. Every sentence earns its place; each covers a distinct concern (what, constraint, fallback, persistence).
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return-value explanation isn't required, freeing the description to focus on behavior. For a 20-param tool it's reasonably complete, though it never acknowledges the translation/summarization features, so an agent must read the schema to discover them.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the input schema fully documents all 20 params; baseline is 3. The description does add real meaning for output_path (persists unsliced transcript, response slimmed) and timestamps, but says nothing about translation, summarization, or language ordering beyond what the schema states.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource: 'Extract YouTube transcripts'. Adds scope details (with timestamps, public captioned videos, fallback to page crawl). It does not explicitly distinguish itself from batch_extract_youtube_transcripts or extract_youtube_comments, so sibling differentiation is left implicit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Works with public captioned videos' gives a constraint, and the output_path sentence implies a persistence workflow. But there is no explicit when-to-use versus the batch or comments variants, and no mention of prerequisites beyond the public-captioned hint.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_search_genresARead-only
Get available search genres for targeted searching.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=false. The description adds no further behavioral details such as whether the genre list is static or dynamic, or any rate limits. It is minimally adequate given the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence with the verb 'Get' front-loaded, containing no unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with no parameters and an output schema, the description sufficiently conveys the tool's purpose. The agent can rely on the output schema for return structure details.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has zero parameters, so the description does not need to explain them. The baseline score of 4 is appropriate as the description does not contradict or fail to add value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves available search genres for targeted searching, using a specific verb ('Get') and resource ('search genres'). It is distinct from sibling tools that perform searches themselves.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when needing genre options for a targeted search, but does not explicitly state when not to use it or provide alternatives among siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_supported_file_formatsARead-only
Get supported file formats (PDF, Office, ZIP) and their capabilities.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations indicate readOnlyHint=true. The description adds context by specifying it returns format capabilities, which goes beyond the annotation. No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence that front-loads the action and examples. No extraneous words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given zero parameters and the presence of an output schema, the description sufficiently informs the agent about the tool's purpose and return value.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are no parameters, so the description does not need to add meaning beyond the input schema. The schema is fully covered.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves supported file formats and their capabilities, with specific examples (PDF, Office, ZIP). It is distinct from sibling tools which focus on crawling, extraction, or processing.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The context is clear: this tool is used to check which formats are supported before using other file-processing tools. No explicit exclusions or alternatives are needed given the tool's simplicity.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_youtube_video_infoARead-only
Get YouTube video metadata and transcript availability. Use output_path to persist the full transcript to disk as markdown and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| timezone | No | IANA timezone name (e.g. 'Asia/Tokyo', 'America/New_York') used to convert the video's publish time, returned under 'published_at'. When unspecified, defaults to 'UTC' (the original YouTube offset is always kept in published_at.original). | UTC |
| llm_model | No | LLM model | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False. | |
| video_url | Yes | YouTube video URL | |
| max_tokens | No | Token limit for summarization | |
| output_path | No | Absolute file path (auto .md extension) to persist the full video info + transcript as markdown. When set, the response is slimmed to metadata+file path. | |
| llm_provider | No | LLM provider | |
| summary_length | No | 'short'|'medium'|'long' | medium |
| include_timestamps | No | Include timestamps | |
| summarize_transcript | No | Summarize transcript | |
| include_content_in_response | No | When True (with output_path set), also include the full transcript in the response. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description usefully discloses the file-persistence side effect and the response-shape trade-off (slim response when output_path is set), which goes beyond the readOnlyHint annotation. It says nothing about the LLM-backed summarization path being invoked by summarize_transcript, which carries latency and cost implications for an agent, and it omits any noting of the tension between a filesystem write and readOnlyHint=true.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences with the core capability front-loaded and the persistence behavior second. Nothing is padded and every clause carries information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With eleven parameters, an output schema, and only a readOnlyHint annotation, the description should convey the tool's full capability set. It never says the tool can summarize the transcript or return full content in the response, leaving two of the most consequential modes to be discovered only by reading the schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all eleven parameters are already documented in the schema, including the slim-response behavior of output_path. The description restates that one behavior but adds no syntax or semantic detail for llm_model, max_tokens, summarize_transcript, summary_length, or include_timestamps, so the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description gives a specific verb and resource ("Get YouTube video metadata and transcript availability") that an agent can act on immediately. However, it never distinguishes this tool from close siblings like extract_youtube_transcript or batch_extract_youtube_transcripts, and it understates its own scope by framing the output as mere "transcript availability" when the schema shows summarization and full-transcript retrieval options.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The second sentence gives one concrete usage rule — set output_path to get a persisted markdown transcript plus a slim response — which is implied guidance on when the parameter matters. But there is no statement of when to prefer this tool over extract_youtube_transcript, no exclusion of cases where it is a poor fit, and no mention of when to enable summarize_transcript.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
intelligent_extractARead-only
Extract specific data from web pages using LLM. Use output_path to persist the full extraction output to disk as JSON and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Target URL | |
| use_llm | No | Enable LLM | |
| llm_model | No | LLM model | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False. | |
| output_path | No | Absolute file path (auto .json extension) to persist the full extracted data + content as JSON. When set, the response is slimmed to metadata+file path (extracted_data.raw_content, content, markdown, table_data removed). | |
| filter_query | No | BM25 filter keywords | |
| llm_provider | No | LLM provider | |
| chunk_content | No | Split content | |
| content_filter | No | 'bm25'|'pruning'|'llm' | bm25 |
| extraction_goal | Yes | Data to extract | |
| custom_instructions | No | LLM instructions | |
| include_content_in_response | No | When True (with output_path set), also keep extracted_data/content in the response. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations indicate readOnlyHint=true, so the tool is non-destructive. The description adds the behavior of output_path enabling slim responses. No further details on rate limits, error handling, or LLM behavior are provided. Does not contradict annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences: first states purpose, second explains key workflow parameter. No extraneous information. Efficient and front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (12 parameters, output schema present), the description covers the main purpose and the critical output_path behavior. It could be improved by clarifying when to use this over crawling or batch tools, but overall it is sufficient for an agent to understand the tool's role.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so most parameter meanings are already clear. The description adds context for output_path, explaining its effect on response structure. This provides marginal additional value beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool extracts specific data from web pages using LLM, distinguishing it from sibling tools like crawl_url which likely return full content. The verb 'extract' and resource 'web pages' are specific.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for extracting data from web pages with optional output persistence, but does not explicitly state when to use this tool versus alternatives like batch_crawl or search_and_crawl. No exclusions or prerequisites are mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
multi_url_crawlBRead-only
Multi-URL crawl with pattern-based config. Max 5 URL patterns per call. Use output_path (directory) to persist full per-URL markdown + index.json; the return shape stays a list, each success item gets an output_file key.
| Name | Required | Description | Default |
|---|---|---|---|
| overwrite | No | Overwrite existing per-URL files inside output_path. Defaults to False. | |
| output_path | No | Absolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names are fine). The list return shape is preserved; each successful item gains an 'output_file' key. Failed items (success=False) are NOT written as .md but still appear in index.json with output_file=null. | |
| base_timeout | No | Timeout per URL (default: 30) | |
| default_config | No | Default config | |
| max_concurrent | No | Has no effect; kept for compatibility. URLs are processed sequentially. | |
| pattern_matching | No | Pattern: 'wildcard' or 'regex' (default: wildcard) | wildcard |
| url_configurations | Yes | URL-config mapping (max 5 URLs). Example: {'https://site1.com': {'wait_for_js': true}} | |
| include_content_in_response | No | When True (with output_path), keep full markdown/content in each list item. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Adds genuine behavioral detail beyond the lone readOnlyHint annotation: the return stays a list and successful items gain an output_file key. However, it omits that URLs are processed sequentially and that output_path causes local filesystem writes (which sits in uneasy tension with readOnlyHint), so the side-effect profile is only partly disclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, front-loaded with the capability and limit, then the persistence behavior. Minor ambiguity from "URL patterns" colliding with the pattern_matching parameter, but nothing is padded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema plus fully described parameters carry most of the load, so the description is adequate. Still missing sibling differentiation and the sequential-processing caveat, which are the two things an agent most needs to make a correct call.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so every parameter already carries its own description, including the 5-URL cap and the output_file behavior. The description restates max 5 URLs and the output_file key without adding format or syntax detail, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb+resource ("Multi-URL crawl") plus scope ("Max 5 URL patterns per call") and the pattern-based config angle. It doesn't name or distinguish itself from the obvious siblings crawl_url, batch_crawl, or deep_crawl_site, so an agent has to infer the difference from the batch/limit constraint alone.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implies usage (batch of up to 5 URLs; use output_path if you want files on disk) but never says when to choose this over batch_crawl or crawl_url, nor any prerequisite/exclusion. With four overlapping crawl siblings, the routing decision is not resolved by the description.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
process_fileBRead-only
Convert PDF, Word, Excel, PowerPoint, ZIP to markdown. Use output_path to persist the full unsliced converted markdown to disk and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | File URL or local path (PDF, Office, ZIP). Supports http/https URLs, file:// URIs, and absolute paths. | |
| llm_model | No | LLM model | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False. | |
| max_size_mb | No | Max file size in MB | |
| output_path | No | Absolute file path (auto .md extension) to persist the full unsliced converted markdown. When set, the response is slimmed to metadata+file path. content_limit/content_offset still affect the response copy but not the on-disk file. | |
| llm_provider | No | LLM provider | |
| content_limit | No | Max characters to return (0=unlimited) | |
| auto_summarize | No | Auto-summarize large content | |
| content_offset | No | Start position for content (0-indexed) | |
| summary_length | No | 'short'|'medium'|'long' | medium |
| include_metadata | No | Include metadata | |
| max_content_tokens | No | Max tokens before summarization | |
| extract_all_from_zip | No | Extract ZIP contents | |
| include_content_in_response | No | When True (with output_path set), keep content in the response too. Note: the response copy is still subject to content_limit/content_offset slicing; only the on-disk file holds the full unsliced payload. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description states the tool 'converts' files, implying a write/mutation operation, but the annotation declares readOnlyHint=true. This is a direct contradiction and severely misleads the agent about the tool's side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences: first states core purpose, second explains key usage pattern. Every word is necessary and front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Essentials covered: conversion types and output_path. With full schema coverage and output schema present, the description is sufficiently complete for a complex tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, baseline 3. The description adds value by explaining the output_path behavior (slim response, full unsliced disk persistence) and slicing effect, going beyond schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool converts PDF, Word, Excel, PowerPoint, ZIP to markdown, and mentions the key output_path feature. It is specific and informative.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus siblings like batch_crawl or crawl_url. The description does not mention alternatives or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_and_crawlARead-only
Search the web (engine as in search_web) and crawl top results. Combines search with full content extraction. Supply output_path (directory) in the request to persist per-page markdown (unsliced) + index.json and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| request | Yes | Dict with: search_query (required), crawl_top_results, search_genre (optional, one of: pdf, documents, presentations, spreadsheets, japanese, english; adds a filetype/language operator to the query, other values return an error), engine (optional, one of: auto, google_custom, brave, exa, searxng; default from SEARCH_ENGINE, auto tries the configured engines in SEARCH_ENGINE_ORDER and moves to the next one on errors other than zero results), result_type (optional; default web; one of web, discussions, faq, videos, news, or several as a list or a comma-separated string; everything except web runs on brave only; news is a separate Brave News Search request; results holds web when requested, else the first type, and the other types go to extra_results; discussions are forum posts with the forum name, answer count, post score, question and top comment; faq are question and answer excerpts; Brave returns these sections for some queries only, and num_results only shortens them), recent_days, generate_markdown, max_content_per_page. Optional persistence keys: output_path (absolute directory — per-page .md files + index.json, the full page bodies are written BEFORE max_content_per_page truncation; dot-containing dir names are fine), include_content_in_response (bool, default False — when True keeps crawled_pages in the response too, still subject to max_content_per_page truncation), overwrite (bool, default False). Failed pages appear in index.json with file=null. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes well beyond readOnlyHint=true by disclosing the filesystem side effect: per-page markdown plus index.json written to output_path, and that the response becomes 'slim' in that mode. It even notes persisted bodies are 'unsliced', i.e. written before max_content_per_page truncation, which is a non-obvious behavioral detail. Missing only permission/overwrite semantics and failure behavior, which the schema covers instead.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three short sentences, front-loaded with the core action and ending with the optional persistence mode, so an agent skimming gets the essential behavior first. The final sentence is dense but every clause carries a distinct fact.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is moderately complex (nested request object, dual search/crawl behavior), the schema is fully documented, and an output schema exists, so return values need not be explained. The description covers the compound behavior and the two response modes, which is what an agent needs to call it correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the nested request description already enumerates every key, including output_path, include_content_in_response, overwrite and failure handling. The description adds modest value by clarifying that output_path is a directory with a specific artifact layout and that persisted content is pre-truncation. That is useful but largely restates schema content, so baseline 3 fits.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific compound action: search the web and crawl the top results, with full content extraction. It anchors the engine parameter to the sibling search_web, so an agent can see this is search_web plus page fetching. It does not explicitly contrast itself with the pure crawl siblings (crawl_url, deep_crawl_site, batch_crawl), leaving the boundary to inference.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'Combines search with full content extraction' implies the condition for use: when snippets alone are insufficient and page bodies are needed. The third sentence describes the persistence mode. However, there is no explicit when-not or named alternative (search_web for listings only, batch_crawl for known URLs), so the routing decision is only implied.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_googleARead-only
Deprecated alias of search_web (same request keys and results; the engine is not limited to Google). Use search_web.
| Name | Required | Description | Default |
|---|---|---|---|
| request | Yes | Dict with: query (required), num_results, search_genre (optional, one of: pdf, documents, presentations, spreadsheets, japanese, english; adds a filetype/language operator to the query, other values return an error), engine (optional, one of: auto, google_custom, brave, exa, searxng; default from SEARCH_ENGINE, auto tries the configured engines in SEARCH_ENGINE_ORDER and moves to the next one on errors other than zero results), result_type (optional; default web; one of web, discussions, faq, videos, news, or several as a list or a comma-separated string; everything except web runs on brave only; news is a separate Brave News Search request; results holds web when requested, else the first type, and the other types go to extra_results; discussions are forum posts with the forum name, answer count, post score, question and top comment; faq are question and answer excerpts; Brave returns these sections for some queries only, and num_results only shortens them), language, region, recent_days, content_limit (int), content_offset (int). Optional persistence keys: output_path (absolute file path, auto .json extension — full unsliced results written to disk BEFORE content_limit/content_offset slicing), include_content_in_response (bool, default False — when True keeps results in the response too, still subject to slicing), overwrite (bool, default False). |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, so the safety profile is covered; the description adds genuinely non-obvious context: the tool is deprecated, and despite its name the engine is not limited to Google. It does not say whether calls emit deprecation warnings or when removal is planned, which would be useful for an agent deciding whether the alias is safe to keep calling.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single sentence that front-loads the critical fact (deprecated) and closes with the actionable directive. Nothing is wasted and nothing important is buried.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a deprecated alias with a fully documented nested schema and an output schema, the description covers what an agent needs: what it is, what it is not (Google-only), and what to use instead. Only the lifecycle details — warnings surfaced, removal timeline — are absent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the nested request object documents every key, including engine and search_genre enums, in detail. The description only asserts 'same request keys and results,' adding no parameter meaning beyond the schema, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description identifies the tool precisely as a deprecated alias of search_web with identical request keys and results, which lets an agent distinguish it from siblings like search_web or batch_search_google. It never states the underlying operation (web search) in its own words, relying on the reader knowing what search_web does, but the alias framing is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives an explicit directive — 'Use search_web' — which is the strongest possible when-to-use guidance for a deprecated tool, and it names the alternative by name. No inference is required about which sibling to prefer.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_webARead-only
Search the web with Google Custom Search, Brave, Exa or SearXNG (engine key, or the server default). Optional search_genre filtering (file types or language; see get_search_genres). Supply output_path in the request to persist the full unsliced result set to disk as JSON and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| request | Yes | Dict with: query (required), num_results, search_genre (optional, one of: pdf, documents, presentations, spreadsheets, japanese, english; adds a filetype/language operator to the query, other values return an error), engine (optional, one of: auto, google_custom, brave, exa, searxng; default from SEARCH_ENGINE, auto tries the configured engines in SEARCH_ENGINE_ORDER and moves to the next one on errors other than zero results), result_type (optional; default web; one of web, discussions, faq, videos, news, or several as a list or a comma-separated string; everything except web runs on brave only; news is a separate Brave News Search request; results holds web when requested, else the first type, and the other types go to extra_results; discussions are forum posts with the forum name, answer count, post score, question and top comment; faq are question and answer excerpts; Brave returns these sections for some queries only, and num_results only shortens them), language, region, recent_days, content_limit (int), content_offset (int). Optional persistence keys: output_path (absolute file path, auto .json extension — full unsliced results written to disk BEFORE content_limit/content_offset slicing), include_content_in_response (bool, default False — when True keeps results in the response too, still subject to slicing), overwrite (bool, default False). |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With readOnlyHint=true covering the safety profile, the description still adds real behavioral detail: output_path writes the full unsliced result set to disk BEFORE content_limit/content_offset slicing and returns a slim response. That persistence side effect is useful context not present in the annotations, though it sits somewhat in tension with readOnlyHint and is not reconciled.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three dense sentences, front-loaded with the core action and back-loaded with the persistence nuance. Nothing is wasted, though the parenthetical engine clause is slightly compressed to the point of ambiguity about what 'engine key' means.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists so return values need no explanation, and the description cross-references get_search_genres for the one enum-like parameter that matters. The main gap is the absence of routing guidance among six search siblings, which the rich schema cannot supply.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline is 3, but the description earns extra credit by surfacing the two highest-leverage parameters and their semantics: search_genre routing to get_search_genres, and output_path plus the pre-slice write ordering. It does not restate engine/result_type behavior, but the schema already does that thoroughly.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource ('Search the web') and immediately names the four backends, which implicitly separates it from the single-engine sibling search_google. It does not explicitly contrast against search_google, batch_search_web, or search_and_crawl, so an agent must infer the boundary, but the multi-engine scope is clear.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage context is implied rather than stated: use search_genre 'see get_search_genres' for file-type or language operators, and add output_path when persistence is needed. There is no explicit when-to-use-this-vs-search_google or batch_search_web guidance despite six closely related siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
12 tool updates
v0.4.2- Changed
batch_crawl1 field changed- changed
Input schema / properties / output_path / descriptionPrevious value: -"Absolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names like /tmp/run.v1 are fine). The list return shape is preserved; each successful item gains an 'output_file' key. Failed items (success=False) are NOT written as .md but still appear in index.json with file=null."New value: +"Absolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names like /tmp/run.v1 are fine). The list return shape is preserved; each successful item gains an 'output_file' key. Failed items (success=False) are NOT written as .md but still appear in index.json with output_file=null."
- Changed
batch_extract_youtube_transcripts1 field changed- changed
Input schema / properties / request / descriptionPrevious value: -"Dict with: urls (max 3), languages, include_timestamps. Optional persistence keys: output_path (absolute directory — per-video .md files + index.json; dot-containing dir names are fine), include_content_in_response (bool; default False — when True, per-video transcripts stay in the response as well), overwrite (bool; default False — existing files rejected). Failed items (success=False) are recorded in index.json with file=null but no .md is written."New value: +"Dict with: urls (max 3), languages, include_timestamps, timezone (IANA name e.g. 'Asia/Tokyo' for converting each video's publish time under metadata.published_at; defaults to 'UTC'). Optional persistence keys: output_path (absolute directory — per-video .md files + index.json; dot-containing dir names are fine), include_content_in_response (bool; default False — when True, per-video transcripts stay in the response as well), overwrite (bool; default False — existing files rejected). Failed items (success=False) are recorded in index.json with file=null but no .md is written."
- Changed
batch_search_google1 field changed- changed
Input schema / properties / request / descriptionPrevious value: -"Dict with: queries (max 3), num_results_per_query, search_genre, recent_days. Optional persistence keys: output_path (absolute file path, auto .json extension — full result set written to disk), include_content_in_response (bool, default False — when True keeps results in the response too), overwrite (bool, default False)."New value: +"Dict with: queries (max 3), num_results_per_query, search_genre (optional, one of: pdf, documents, presentations, spreadsheets, japanese, english; adds a filetype/language operator to the query, other values return an error), engine (optional, one of: auto, google_custom, brave, exa, searxng; default from SEARCH_ENGINE, auto tries the configured engines in SEARCH_ENGINE_ORDER and moves to the next one on errors other than zero results), result_type (optional; default web; one of web, discussions, faq, videos, news, or several as a list or a comma-separated string; everything except web runs on brave only; news is a separate Brave News Search request; results holds web when requested, else the first type, and the other types go to extra_results; discussions are forum posts with the forum name, answer count, post score, question and top comment; faq are question and answer excerpts; Brave returns these sections for some queries only, and num_results only shortens them), recent_days. Optional persistence keys: output_path (absolute file path, auto .json extension — full result set written to disk), include_content_in_response (bool, default False — when True keeps results in the response too), overwrite (bool, default False)."
- Added
batch_search_web - Changed
deep_crawl_site2 fields changed- changed
Input schema / properties / max_depth / descriptionPrevious value: -"Link depth (1-2)"New value: +"Link depth, 1-2 (default: 2). Values outside the range return an error." - changed
Input schema / properties / max_pages / descriptionPrevious value: -"Max pages (max: 10)"New value: +"Max pages, 1-10 (default: 5). Values outside the range return an error."
- Changed
extract_entities2 fields changed- changed
Input schema / properties / custom_patterns / descriptionPrevious value: -"Custom regex patterns"New value: +"Custom regex patterns as {type_name: regex}; list the type_name in entity_types to use it (regex mode only)" - changed
Input schema / properties / entity_types / descriptionPrevious value: -"Types: email, phone, url, date, ip, price"New value: +"Regex types: emails, phones, urls, dates, ips, prices, credit_cards, coordinates (singular forms like email/phone are accepted), plus any custom_patterns keys. Without use_llm, other types are listed under unsupported_entity_types with a warning. With use_llm=True any type name works (e.g. people, organizations, locations, products, events)."
- Changed
extract_youtube_transcript1 field changed- added
Input schema / properties / timezoneAdded value: +{ + "default": "UTC", + "description": "IANA timezone name (e.g. 'Asia/Tokyo', 'America/New_York') used to convert the video's publish time, returned under metadata.published_at (requires enrich_metadata=True). When unspecified, defaults to 'UTC' (the original YouTube offset is always kept in metadata.published_at.original).", + "type": "string" +}
- Changed
get_youtube_video_info1 field changed- added
Input schema / properties / timezoneAdded value: +{ + "default": "UTC", + "description": "IANA timezone name (e.g. 'Asia/Tokyo', 'America/New_York') used to convert the video's publish time, returned under 'published_at'. When unspecified, defaults to 'UTC' (the original YouTube offset is always kept in published_at.original).", + "type": "string" +}
- Changed
multi_url_crawl2 fields changed- changed
Input schema / properties / max_concurrent / descriptionPrevious value: -"Max concurrent (default: 3)"New value: +"Has no effect; kept for compatibility. URLs are processed sequentially." - changed
Input schema / properties / output_path / descriptionPrevious value: -"Absolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names are fine). The list return shape is preserved; each successful item gains an 'output_file' key. Failed items (success=False) are NOT written as .md but still appear in index.json with file=null."New value: +"Absolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names are fine). The list return shape is preserved; each successful item gains an 'output_file' key. Failed items (success=False) are NOT written as .md but still appear in index.json with output_file=null."
- Changed
search_and_crawl1 field changed- changed
Input schema / properties / request / descriptionPrevious value: -"Dict with: search_query (required), crawl_top_results, search_genre, recent_days, generate_markdown, max_content_per_page. Optional persistence keys: output_path (absolute directory — per-page .md files + index.json, the full page bodies are written BEFORE max_content_per_page truncation; dot-containing dir names are fine), include_content_in_response (bool, default False — when True keeps crawled_pages in the response too, still subject to max_content_per_page truncation), overwrite (bool, default False). Failed pages appear in index.json with file=null."New value: +"Dict with: search_query (required), crawl_top_results, search_genre (optional, one of: pdf, documents, presentations, spreadsheets, japanese, english; adds a filetype/language operator to the query, other values return an error), engine (optional, one of: auto, google_custom, brave, exa, searxng; default from SEARCH_ENGINE, auto tries the configured engines in SEARCH_ENGINE_ORDER and moves to the next one on errors other than zero results), result_type (optional; default web; one of web, discussions, faq, videos, news, or several as a list or a comma-separated string; everything except web runs on brave only; news is a separate Brave News Search request; results holds web when requested, else the first type, and the other types go to extra_results; discussions are forum posts with the forum name, answer count, post score, question and top comment; faq are question and answer excerpts; Brave returns these sections for some queries only, and num_results only shortens them), recent_days, generate_markdown, max_content_per_page. Optional persistence keys: output_path (absolute directory — per-page .md files + index.json, the full page bodies are written BEFORE max_content_per_page truncation; dot-containing dir names are fine), include_content_in_response (bool, default False — when True keeps crawled_pages in the response too, still subject to max_content_per_page truncation), overwrite (bool, default False). Failed pages appear in index.json with file=null."
- Changed
search_google1 field changed- changed
Input schema / properties / request / descriptionPrevious value: -"Dict with: query (required), num_results, search_genre, language, region, recent_days, content_limit (int), content_offset (int). Optional persistence keys: output_path (absolute file path, auto .json extension — full unsliced results written to disk BEFORE content_limit/content_offset slicing), include_content_in_response (bool, default False — when True keeps results in the response too, still subject to slicing), overwrite (bool, default False)."New value: +"Dict with: query (required), num_results, search_genre (optional, one of: pdf, documents, presentations, spreadsheets, japanese, english; adds a filetype/language operator to the query, other values return an error), engine (optional, one of: auto, google_custom, brave, exa, searxng; default from SEARCH_ENGINE, auto tries the configured engines in SEARCH_ENGINE_ORDER and moves to the next one on errors other than zero results), result_type (optional; default web; one of web, discussions, faq, videos, news, or several as a list or a comma-separated string; everything except web runs on brave only; news is a separate Brave News Search request; results holds web when requested, else the first type, and the other types go to extra_results; discussions are forum posts with the forum name, answer count, post score, question and top comment; faq are question and answer excerpts; Brave returns these sections for some queries only, and num_results only shortens them), language, region, recent_days, content_limit (int), content_offset (int). Optional persistence keys: output_path (absolute file path, auto .json extension — full unsliced results written to disk BEFORE content_limit/content_offset slicing), include_content_in_response (bool, default False — when True keeps results in the response too, still subject to slicing), overwrite (bool, default False)."
- Added
search_web
19 tool updates
v0.3.2- Added
batch_crawl - Added
batch_extract_youtube_transcripts - Added
batch_search_google - Added
crawl_url - Added
crawl_url_with_fallback - Added
deep_crawl_site - Added
enhanced_process_large_content - Added
extract_entities - Added
extract_structured_data - Added
extract_youtube_comments - Added
extract_youtube_transcript - Added
get_search_genres - Added
get_supported_file_formats - Added
get_youtube_video_info - Added
intelligent_extract - Added
multi_url_crawl - Added
process_file - Added
search_and_crawl - Added
search_google
13 tool updates
v0.1.6- Removed
batch_crawl - Removed
batch_extract_youtube_transcripts - Removed
batch_search_google - Removed
crawl_url - Removed
crawl_url_with_fallback - Removed
deep_crawl_site - Removed
extract_structured_data - Removed
extract_youtube_transcript - Removed
get_youtube_video_info - Removed
multi_url_crawl - Removed
process_file - Removed
search_and_crawl - Removed
search_google
13 tool updates
v0.1.7- First observed
batch_crawl - First observed
batch_extract_youtube_transcripts - First observed
batch_search_google - First observed
crawl_url - First observed
crawl_url_with_fallback - First observed
deep_crawl_site - First observed
extract_structured_data - First observed
extract_youtube_transcript - First observed
get_youtube_video_info - First observed
multi_url_crawl - First observed
process_file - First observed
search_and_crawl - First observed
search_google
TDQS
Scored across 21 tools
Several tools have overlapping boundaries: crawl_url vs crawl_url_with_fallback (same operation with fallback strategies), batch_crawl vs multi_url_crawl (both crawl multiple URLs, differing only in limits/config style), and extract_structured_data vs intelligent_extract (both LLM-based extraction). The deprecated search_google/batch_search_google aliases add further surface duplication, though descriptions do flag them as aliases and help disambiguate.
All names use snake_case and mostly follow a verb_noun pattern (crawl_url, extract_entities, search_web, get_youtube_video_info). A few deviate to adjective-first phrasing (enhanced_process_large_content, intelligent_extract), which is a minor inconsistency but still readable and predictable.
21 tools is on the heavy side for a crawling/search server, and the count is inflated by two deprecated aliases plus redundant crawl variants (batch_crawl/multi_url_crawl, crawl_url/crawl_url_with_fallback). The core capability set is reasonable, but pruning the aliases and merged variants would tighten the surface.
Coverage is broad and coherent: search (single/batch/genres), crawling (single, batch, multi-URL, deep site, fallback), extraction (entities, structured data, LLM), YouTube (info, transcript, comments, batch), and file conversion, plus format/genre discovery tools. Minor gaps like sitemap/robots handling or cache/session management exist but agents can work around them.
Maintenance
Related MCP Connectors
Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
One MCP server for all web data. Scrape, search, crawl, and AI research in one install.
An MCP server that gives your AI access to the source code and docs of all public github repos
MCP server for OpenAI API (chat completions, image generation, embeddings) via AceDataCloud
Related MCP Servers
- FlicenseBqualityDmaintenanceAn MCP Server for Web scraping and Crawling, built using Crawl4AI224-
- FlicenseNot gradedqualityDmaintenanceA locally-hosted MCP server that provides AI assistants with advanced web crawling capabilities, including structured data extraction, deep site crawling, and page screenshots. It enables users to convert single or multiple URLs into clean Markdown content for processing by LLMs without requiring external API keys for basic features.-
- AlicenseAqualityBmaintenanceWeb extraction MCP server for AI agents. Extract structured data from any URL with built-in Cloudflare bypass, JavaScript rendering, and intelligent parsing. Returns clean markdown or JSON.5794 npm2MIT
- AlicenseNot gradedqualityCmaintenanceMCP server integrating Crawl4AI for universal web crawling and data extraction. Enables AI agents to crawl, extract markdown/HTML, take screenshots, generate PDFs, and execute JavaScript on web pages.51 npm7MIT