Crawl-MCP
Crawl-MCP: Servidor MCP no oficial para crawl4ai
⚠️ Importante: Esta es una implementación de servidor MCP no oficial para la excelente biblioteca crawl4ai. No está afiliado con el proyecto original crawl4ai.
Un servidor completo del Protocolo de Contexto de Modelo (MCP) que envuelve la potente biblioteca crawl4ai con capacidades avanzadas de IA. Extrae y analiza contenido de cualquier fuente: páginas web, archivos PDF, documentos de Office, vídeos de YouTube y más. Incluye resumen inteligente para reducir drásticamente el uso de tokens mientras se conserva la información esencial.
🌟 Características principales
🔍 Integración con la búsqueda de Google - 7 géneros de búsqueda optimizados con operadores oficiales de Google
🔍 Rastreo web avanzado: soporte para JavaScript, mapeo profundo de sitios, extracción de entidades
🌐 Extracción universal de contenido: páginas web, archivos PDF, documentos de Word, Excel, PowerPoint, archivos ZIP
🤖 Resumen potenciado por IA: reducción inteligente de tokens (hasta un 88,5%) mientras se conserva la información esencial
🎬 Integración con YouTube: extrae transcripciones y resúmenes de vídeos sin necesidad de claves API
⚡ Listo para producción: 19 herramientas especializadas con gestión integral de errores
Related MCP server: Crawl4AI MCP Server
🚀 Inicio rápido
Requisitos previos (necesarios primero)
Python 3.11 o superior (FastMCP requiere Python 3.11+)
Instalar dependencias del sistema para Playwright:
Ubuntu 24.04 LTS (requiere manual):
# Manual setup required due to t64 library transition
sudo apt update && sudo apt install -y \
libnss3 libatk-bridge2.0-0 libxss1 libasound2t64 \
libgbm1 libgtk-3-0t64 libxshmfence-dev libxrandr2 \
libxcomposite1 libxcursor1 libxdamage1 libxi6 \
fonts-noto-color-emoji fonts-unifont python3-venv python3-pip
python3 -m venv venv && source venv/bin/activate
pip install playwright==1.55.0 && playwright install chromium
sudo playwright install-depsOtros Linux/macOS:
sudo bash scripts/prepare_for_uvx_playwright.shWindows (como administrador):
scripts/prepare_for_uvx_playwright.ps1Instalación
UVX (Recomendado - El más sencillo):
# After system preparation above - that's it!
uvx --from git+https://github.com/walksoda/crawl-mcp crawl-mcpDocker (Listo para producción):
# Clone the repository
git clone https://github.com/walksoda/crawl-mcp
cd crawl-mcp
# Build and run with Docker Compose (STDIO mode)
docker-compose up --build
# Or build and run HTTP mode on port 8000
docker-compose --profile http up --build crawl4ai-mcp-http
# Or build manually
docker build -t crawl4ai-mcp .
docker run -it crawl4ai-mcpCaracterísticas de Docker:
🔧 Soporte para múltiples navegadores: navegadores Chromium, Firefox y Webkit en modo headless
🐧 Google Chrome: Chrome Stable adicional para compatibilidad
⚡ Rendimiento optimizado: flags de navegador preconfiguradas para Docker
🔒 Seguridad: ejecución con usuario no root
📦 Dependencias completas: todas las bibliotecas necesarias incluidas
Configuración de Claude Desktop
Instalación con UVX:
Agregue esto a su claude_desktop_config.json:
{
"mcpServers": {
"crawl-mcp": {
"transport": "stdio",
"command": "uvx",
"args": [
"--from",
"git+https://github.com/walksoda/crawl-mcp",
"crawl-mcp"
],
"env": {
"CRAWL4AI_LANG": "en"
}
}
}
}Modo HTTP de Docker:
{
"mcpServers": {
"crawl-mcp": {
"transport": "http",
"baseUrl": "http://localhost:8000"
}
}
}Para interfaz en japonés:
"env": {
"CRAWL4AI_LANG": "ja"
}📖 Documentación
Tema | Descripción |
Instrucciones completas de instalación para todas las plataformas | |
Documentación completa de herramientas y ejemplos de uso | |
Configuraciones de configuración específicas de la plataforma | |
Acceso a la API HTTP y métodos de integración | |
Técnicas y flujos de trabajo para usuarios avanzados | |
Contribución y configuración de desarrollo |
Documentación específica por idioma
🛠️ Resumen de herramientas
Rastreo web (3)
crawl_url- Extrae contenido de páginas web con soporte para JavaScriptdeep_crawl_site- Rastrea múltiples páginas de un sitio con profundidad configurablecrawl_url_with_fallback- Rastrea con estrategias de respaldo para sitios anti-bot
Extracción de datos (3)
intelligent_extract- Extrae datos específicos de páginas web usando LLMextract_entities- Extrae entidades (correos electrónicos, teléfonos, etc.) de páginas webextract_structured_data- Extrae datos estructurados usando selectores CSS o LLM
YouTube (4)
extract_youtube_transcript- Extrae transcripciones de YouTube con marcas de tiempobatch_extract_youtube_transcripts- Extrae transcripciones de múltiples vídeos de YouTube (máx. 3)get_youtube_video_info- Obtiene metadatos de vídeos de YouTube y disponibilidad de transcripcionesextract_youtube_comments- Extrae comentarios de vídeos de YouTube con paginación
Búsqueda (4)
search_google- Busca en Google con filtrado de génerobatch_search_google- Realiza múltiples búsquedas en Google (máx. 3)search_and_crawl- Busca en Google y rastrea los mejores resultadosget_search_genres- Obtiene los géneros de búsqueda disponibles
Procesamiento de archivos (3)
process_file- Convierte PDF, Word, Excel, PowerPoint, ZIP a markdownget_supported_file_formats- Obtiene los formatos de archivo y capacidades compatiblesenhanced_process_large_content- Procesa contenido grande con fragmentación y filtrado BM25
Operaciones por lotes (2)
batch_crawl- Rastrea múltiples URLs con respaldo (máx. 3 URLs)multi_url_crawl- Rastreo de múltiples URLs con configuración basada en patrones (máx. 5 patrones de URL)
💾 Persistir resultados grandes en disco (ahorrador de tokens)
Todas las herramientas de recopilación de información aceptan un parámetro opcional output_path que escribe el contenido completo obtenido directamente en el disco y devuelve una respuesta ligera solo con metadatos. Esto permite que un LLM obtenga páginas enormes, transcripciones largas de YouTube o lotes completos sin agotar su presupuesto de contexto; lea desde el archivo guardado solo cuando sea necesario.
Cómo funciona:
Las herramientas de un solo archivo (p. ej.,
crawl_url,extract_youtube_transcript) escriben un.md(o.jsonpara herramientas de tipo JSON); pase una ruta de archivo absoluta; la extensión se añade automáticamente si se omite. Se rechaza un archivo normal existente en esa ruta a menos queoverwrite=true.Las herramientas por lotes (
batch_crawl,multi_url_crawl,deep_crawl_site,search_and_crawl,batch_extract_youtube_transcripts) esperan una ruta de directorio absoluta y escriben un.mdpor URL más unindex.json. Cualquier ruta inexistente se trata como un directorio y se crea, incluidos los nombres que contienen puntos como/tmp/run.v1. Si la ruta ya existe como un archivo normal, la llamada se rechaza.batch_crawl/multi_url_crawlmantienen su forma de retorno deliste incrustan una claveoutput_fileen cada elemento exitoso.Las herramientas de diccionario de solicitud (
search_google,batch_search_google,search_and_crawl,batch_extract_youtube_transcripts) leen las claves de persistencia directamente de su diccionario de solicitud.Parámetros comunes:
output_path(absoluto;Noneo""omite la persistencia),include_content_in_response(predeterminadofalse; cuando estrue, el contenido también se incluye en la respuesta, todavía sujeto a cualquier recorte decontent_limit/content_offset/max_content_per_page),overwrite(predeterminadofalse).Las escrituras son atómicas por archivo (archivo temporal +
os.replace); los directorios principales se crean automáticamente; la carga útil completa sin recortar se persiste antes de cualquier recorte o truncamiento interno de la herramienta, por lo que la copia en disco siempre está completa incluso cuando la respuesta está recortada.Las herramientas de diccionario por lotes (
deep_crawl_site,search_and_crawl,batch_extract_youtube_transcripts) omiten la persistencia por elemento para aquellos que informansuccess=false; estos aún aparecen enindex.jsonconfile: nullpara que los llamadores puedan razonar sobre la lista de intentos.
Ejemplo de archivo único Markdown:
{
"tool": "crawl_url",
"arguments": {
"url": "https://example.com/long-article",
"output_path": "/tmp/crawl_out/article.md"
}
}Extracción estructurada JSON (extensión añadida automáticamente):
{
"tool": "extract_structured_data",
"arguments": {
"url": "https://example.com/products",
"extraction_type": "css",
"css_selectors": {"price": ".price", "name": "h1"},
"output_path": "/tmp/crawl_out/products"
}
}Modo de directorio por lotes:
{
"tool": "batch_crawl",
"arguments": {
"urls": ["https://a.example", "https://b.example"],
"output_path": "/tmp/crawl_out/batch_run1"
}
}Cada archivo markdown persistido comienza con un bloque YAML frontmatter que contiene url, title, fetched_at y source_tool para que el artefacto sea autodescriptivo.
🎯 Casos de uso comunes
Investigación de contenido:
search_and_crawl → extract_structured_data → analysisMinería de documentación:
deep_crawl_site → batch processing → extractionAnálisis de medios:
extract_youtube_transcript → summarization workflowMapeo de sitios:
batch_crawl → multi_url_crawl → comprehensive data🚨 Solución de problemas rápida
Problemas de instalación:
Vuelva a ejecutar los scripts de configuración con los privilegios adecuados
Pruebe el método de instalación de desarrollo
Compruebe que las dependencias del navegador estén instaladas
Problemas de rendimiento:
Use
wait_for_js: truepara sitios con mucho JavaScriptAumente el tiempo de espera para páginas de carga lenta
Use
extract_structured_datapara una extracción dirigida
Problemas de configuración:
Compruebe la sintaxis JSON en
claude_desktop_config.jsonVerifique que las rutas de archivo sean absolutas
Reinicie Claude Desktop después de los cambios de configuración
🏗️ Estructura del proyecto
Biblioteca original: crawl4ai por unclecode
Envoltorio MCP: Este repositorio (walksoda)
Implementación: Integración de terceros no oficial
📄 Licencia
Este proyecto es un envoltorio no oficial de la biblioteca crawl4ai. Consulte la licencia original de crawl4ai para conocer la funcionalidad subyacente.
🤝 Contribución
Consulte nuestra Guía de desarrollo para conocer las pautas de contribución y las instrucciones de configuración de desarrollo.
🔗 Proyectos relacionados
crawl4ai - La biblioteca de rastreo web subyacente
Model Context Protocol - El estándar que implementa este servidor
Claude Desktop - Cliente principal para servidores MCP
Available Tools
19 toolsbatch_crawlA
Crawl multiple URLs with fallback. Max 3 URLs per call. Use output_path (directory) to persist full per-URL markdown + index.json; the return shape stays a list, each success item gets an output_file key.
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | URLs to crawl (max 3) | |
| base_timeout | No | Timeout per URL (default: 30) | |
| generate_markdown | No | Generate markdown (default: True) | |
| extract_media | No | Extract media (default: False) | |
| wait_for_js | No | Wait for JS (default: False) | |
| output_path | No | Absolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names like /tmp/run.v1 are fine). The list return shape is preserved; each successful item gains an 'output_file' key. Failed items (success=False) are NOT written as .md but still appear in index.json with file=null. | |
| include_content_in_response | No | When True (with output_path), keep full markdown/content in each list item. Defaults to False so the response stays token-efficient. | |
| overwrite | No | Overwrite existing per-URL files inside output_path. Defaults to False (existing files cause an output_path_exists error, returned as a single-element list). |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Without annotations, the description covers key behaviors: output_path creation and validation, per-URL file writing, error handling for failures and existing files, and the effect of include_content_in_response. It does not mention authentication or rate limits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, compact paragraph that front-loads the core purpose and constraints. It packs significant detail without redundancy, though it could benefit from slight restructuring for readability.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (8 parameters, output schema present), the description covers most behavioral aspects and parameter interactions. A minor gap is the lack of elaboration on the fallback mechanism mentioned in the title.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, but the description adds meaningful context for output_path (e.g., dot-containing names allowed, index.json handling) and include_content_in_response (token efficiency). This goes beyond the schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool crawls multiple URLs with fallback and specifies a maximum of 3 URLs per call. It identifies the resource and action, but does not explicitly differentiate from siblings like multi_url_crawl or crawl_url_with_fallback.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides guidance on using output_path for persistence and notes constraints like maximum URLs. However, it lacks explicit context on when to choose this tool over alternatives or when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
batch_extract_youtube_transcriptsBRead-only
Extract transcripts from multiple YouTube videos. Max 3 URLs per call. Supply output_path (directory) in the request to persist per-video markdown files + index.json and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| request | Yes | Dict with: urls (max 3), languages, include_timestamps. Optional persistence keys: output_path (absolute directory — per-video .md files + index.json; dot-containing dir names are fine), include_content_in_response (bool; default False — when True, per-video transcripts stay in the response as well), overwrite (bool; default False — existing files rejected). Failed items (success=False) are recorded in index.json with file=null but no .md is written. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotation declares readOnlyHint=true, implying no state changes, but description and schema describe writing .md files and index.json to disk, which modifies the filesystem. This contradiction is severe, and per rubric, score is 1.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences front-load the purpose and state key constraints (max URLs, output_path requirement) without extraneous information. Every word is necessary and no fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite the annotation contradiction, the description covers purpose, capacity, and persistence requirement. The schema fills in parameter details, and output schema exists. Missing some behavioral details like error handling for failed items (mentioned only in schema), but overall sufficient for invocation given the rich schema.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, and the parameter description in the schema already details all keys (urls, languages, output_path, etc.). The tool description adds minimal value beyond what the schema provides, so baseline 3 is appropriate.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the tool extracts transcripts from multiple YouTube videos, distinguishing it from the singular counterpart extract_youtube_transcript. It specifies a max of 3 URLs per call and mentions persistence, which is specific and actionable.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Description provides constraints (max 3 URLs, need output_path) but does not explicitly guide when to use this batch tool versus the singular extract_youtube_transcript or other sibling tools. Usage context is implied but exclusions and alternatives are not stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
batch_search_googleARead-only
Perform multiple Google searches. Max 3 queries per call. Supply output_path in the request to persist the full result set to disk as JSON and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| request | Yes | Dict with: queries (max 3), num_results_per_query, search_genre, recent_days. Optional persistence keys: output_path (absolute file path, auto .json extension — full result set written to disk), include_content_in_response (bool, default False — when True keeps results in the response too), overwrite (bool, default False). |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations provide readOnlyHint=true, consistent with search. Description adds persistence behavior (writing to disk) and slim response, but lacks details on error handling or quota implications. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences pack the core action, limit, and optional persistence, with no redundancy. Front-loaded with purpose and constraint.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given rich schema descriptions and output schema existence, the description adequately conveys the core function. Missing behavioral details like error responses or handling of partial failures, but sufficient for typical use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, with detailed parameter descriptions. The main description adds minimal extra value beyond schema, achieving baseline for a fully covered schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states it performs multiple Google searches with a max of 3 queries, clearly distinguishing it from single-query tools like search_google and crawling tools like batch_crawl.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description indicates when to use (multiple searches) but lacks explicit when-not-to-use or sibling comparisons. It implicitly suggests use for batch queries over single search but could be more direct.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_urlA
Extract web page content with JavaScript support. Use wait_for_js=true for SPAs. Use content_offset/content_limit to paginate the response. Use output_path to persist the full unsliced content to disk as markdown and receive a slim metadata-only response.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to crawl | |
| css_selector | No | CSS selector for extraction | |
| extract_media | No | Extract images/videos | |
| take_screenshot | No | Take screenshot | |
| generate_markdown | No | Generate markdown | |
| include_cleaned_html | No | Include cleaned HTML | |
| wait_for_selector | No | Wait for element to load | |
| timeout | No | Timeout in seconds | |
| wait_for_js | No | Wait for JavaScript | |
| auto_summarize | No | Auto-summarize large content | |
| use_undetected_browser | No | Bypass bot detection | |
| content_limit | No | Max characters to return (0=unlimited) | |
| content_offset | No | Start position for content (0-indexed) | |
| output_path | No | Absolute file path (auto .md extension) to persist the full unsliced markdown. When set, the response is slimmed to metadata+file path to save tokens. content_limit/content_offset still affect the response copy but not the on-disk file. | |
| include_content_in_response | No | When True (with output_path set), keep markdown/content in the response too. Note: the response copy is still subject to content_limit/content_offset slicing; only the on-disk file holds the full unsliced payload. Defaults to False. | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False (existing files are rejected before any fetch). |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description adds significant behavioral detail: how output_path slimdown works, the interaction between content_offset/limit and the on-disk file, and the include_content_in_response flag. This goes beyond basic functionality.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise: three short sentences that front-load the purpose and quickly cover the most important usage scenarios. No unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 16 parameters and an output schema, the description covers the most important behaviors (JS, pagination, persistence) but omits details on less common params like css_selector or extract_media. Still, the core agent needs are addressed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description adds value by explaining the interplay between output_path, content_offset, and content_limit, and advises when to use wait_for_js, which is not present in the schema alone.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool extracts web page content with JavaScript support, and provides specific use cases like SPAs and pagination. However, it does not differentiate from sibling tools like 'crawl_url_with_fallback' or 'deep_crawl_site', which lowers the score slightly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit usage guidelines for key features (wait_for_js for SPAs, pagination with offsets, persisting content to disk). It lacks guidance on when not to use this tool or alternatives, but provides enough context for common scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawl_url_with_fallbackARead-only
Crawl with fallback strategies for anti-bot sites. Use content_offset/content_limit to paginate the response. Use output_path to persist the full unsliced content to disk as markdown and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to crawl | |
| css_selector | No | CSS selector | |
| extract_media | No | Extract media | |
| take_screenshot | No | Take screenshot | |
| generate_markdown | No | Generate markdown | |
| wait_for_selector | No | Element to wait for | |
| timeout | No | Timeout in seconds | |
| wait_for_js | No | Wait for JavaScript | |
| auto_summarize | No | Auto-summarize content | |
| content_limit | No | Max characters to return (0=unlimited) | |
| content_offset | No | Start position for content (0-indexed) | |
| output_path | No | Absolute file path (auto .md extension) to persist the full unsliced markdown. When set, the response is slimmed to metadata+file path. content_limit/content_offset still affect the response copy but not the on-disk file. | |
| include_content_in_response | No | When True (with output_path set), keep markdown/content in the response too. Note: the response copy is still subject to content_limit/content_offset slicing; only the on-disk file holds the full unsliced payload. | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False (existing files rejected before any fetch). |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate readOnlyHint=true, which is consistent with crawling. Description adds that fallback strategies exist for anti-bot sites, but does not detail what those strategies are or any other behavioral traits.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with core purpose, no fluff. Each sentence adds distinct value: purpose, pagination, and persistence.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 14 parameters and an output schema (not shown), the description covers the main behavioral aspects (fallback, pagination, persistence). Minor gap: no explanation of the fallback strategies themselves, but overall adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 100% schema coverage, baseline is 3. The description adds meaningful context for pagination (content_offset/content_limit) and persistence (output_path with behavior rules), going beyond the schema comments.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action ('Crawl') and target ('anti-bot sites') with fallback strategies, but does not differentiate from the sibling tool 'crawl_url' which likely has similar purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage via mention of pagination and persistence, but does not provide explicit guidance on when to choose this tool over alternatives like 'crawl_url' or 'search_and_crawl'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
deep_crawl_siteARead-only
Crawl multiple pages from a site with configurable depth. Use output_path (directory) to persist per-URL markdown files + index.json; the response is then slimmed to metadata only.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Starting URL | |
| max_depth | No | Link depth (1-2) | |
| max_pages | No | Max pages (max: 10) | |
| crawl_strategy | No | 'bfs'|'dfs'|'best_first' | bfs |
| include_external | No | Follow external links | |
| url_pattern | No | URL filter pattern | |
| score_threshold | No | Min relevance 0-1 | |
| extract_media | No | Extract media | |
| base_timeout | No | Timeout per page | |
| output_path | No | Absolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names like /tmp/run.v1 are fine). When set, the response is slimmed to metadata+file paths. Failed items (success=False) are NOT written as .md but still recorded in index.json with file=null. | |
| include_content_in_response | No | When True (with output_path set), also include per-page content/markdown in the response items. Defaults to False so the response stays token-efficient. | |
| overwrite | No | Overwrite existing per-URL files inside output_path. Defaults to False (existing files cause an output_path_exists error). |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses that writing files to output_path is a side effect, and that the response is slimmed when output_path is set. This adds value beyond the readOnlyHint annotation, but it does not cover all behavioral traits (e.g., rate limits, auth).
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with purpose and key behavior. Every sentence earns its place; no redundancy with schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given tool complexity (12 parameters) and presence of output schema, the description is fairly complete. It explains the main output_path feature and behavior, but does not elaborate on other parameters or when to use depth vs alternatives.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description adds significant meaning to parameters like output_path, include_content_in_response, and overwrite, explaining their effects on response and file persistence. Schema coverage is 100% but the description enriches understanding.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it crawls multiple pages from a site with configurable depth, using a verb+resource structure. It distinguishes from siblings like crawl_url by mentioning multi-page and output persistence.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for multi-page crawling with depth, but does not explicitly state when to use this tool versus alternatives like crawl_url or batch_crawl. No when-not or exclusions are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
enhanced_process_large_contentBRead-only
Process large content with chunking and BM25 filtering. Use output_path to persist chunks + summaries to disk as JSON and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to process | |
| chunking_strategy | No | 'topic'|'sentence'|'overlap'|'regex' | sentence |
| filtering_strategy | No | 'bm25'|'pruning'|'llm' | bm25 |
| filter_query | No | Keywords for BM25 filtering | |
| max_chunk_tokens | No | Max tokens per chunk | |
| chunk_overlap | No | Overlap tokens | |
| extract_top_chunks | No | Top chunks to extract | |
| similarity_threshold | No | Min similarity 0-1 | |
| summarize_chunks | No | Summarize chunks | |
| merge_strategy | No | 'hierarchical'|'linear' | linear |
| final_summary_length | No | 'short'|'medium'|'long' | short |
| output_path | No | Absolute file path (auto .json extension) to persist the full chunks + summaries as JSON. When set, the response is slimmed to metadata+file path (chunks, chunk_summaries, merged_summary, final_summary removed). | |
| include_content_in_response | No | When True (with output_path set), also include chunks/summaries in the response. Defaults to False. | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations declare readOnlyHint=true, but the description states the tool persists data to disk, which is a write operation. This is a direct contradiction. Additionally, the description does not disclose other behavioral traits such as required permissions, rate limits, or side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences long, directly stating the core functionality and key usage advice. Every word earns its place with no fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's 14 parameters and existence of an output schema, the description is minimal but covers the main point. However, it lacks details about return structure, error conditions, and behavioral nuances beyond the output_path feature.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Input schema has 100% coverage for all 14 parameters, providing clear descriptions. The description adds value by explaining the effect of output_path and the slim response, which enhances understanding beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it processes large content with chunking and BM25 filtering, and mentions persisting to disk. It distinguishes itself from siblings by specifying the advanced processing features, though it doesn't explicitly differentiate from similar tools like 'process_file'.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. It only implies usage via the description but lacks explicit context, exclusions, or comparisons with sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_entitiesARead-only
Extract entities (emails, phones, etc.) from web pages. Use output_path to persist the full entity extraction output to disk as JSON and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Target URL | |
| entity_types | Yes | Types: email, phone, url, date, ip, price | |
| custom_patterns | No | Custom regex patterns | |
| include_context | No | Include context | |
| deduplicate | No | Remove duplicates | |
| use_llm | No | Use LLM for NER | |
| llm_provider | No | LLM provider | |
| llm_model | No | LLM model | |
| output_path | No | Absolute file path (auto .json extension) to persist the full entity extraction as JSON. When set, the response is slimmed (content, markdown, extracted_data.raw_content removed). | |
| include_content_in_response | No | When True (with output_path set), also keep the entity data in the response. Defaults to False. | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Description adds context beyond readOnlyHint annotation, detailing the output_path parameter behavior (persisting full output to disk and returning slim response). No contradictions with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with purpose, no wasted words. Efficient and well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Tool has output schema covering return values, and description explains key behavioral nuance. For a complex 11-parameter tool, description is brief but adequate given schema richness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so baseline 3 is appropriate. Description enhances understanding of the output_path parameter, explaining its effect on response format, which exceeds baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states 'Extract entities (emails, phones, etc.) from web pages,' providing a specific verb and resource with examples. This distinguishes it from siblings like extract_structured_data and intelligent_extract.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Description does not provide explicit when-to-use or when-not-to-use guidance. No mention of alternatives or context for selecting this tool over similar siblings like extract_structured_data.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_structured_dataBRead-only
Extract structured data using CSS selectors or LLM. Use output_path to persist the full extraction (including table_data) to disk as JSON and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Target URL | |
| extraction_type | No | 'css'|'llm'|'table' | css |
| css_selectors | No | CSS selector mapping | |
| extraction_schema | No | Schema definition | |
| generate_markdown | No | Generate markdown | |
| wait_for_js | No | Wait for JavaScript | |
| timeout | No | Timeout in seconds | |
| use_llm_table_extraction | No | Use LLM table extraction | |
| table_chunking_strategy | No | 'intelligent'|'fixed'|'semantic' | intelligent |
| output_path | No | Absolute file path (auto .json extension) to persist the full extracted_data + table_data as JSON. When set, the response is slimmed (content, markdown, table_data, extracted_data.raw_content removed). | |
| include_content_in_response | No | When True (with output_path set), also keep extracted_data/table_data/content in the response. Defaults to False. | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations indicate readOnlyHint=true, which matches the extraction purpose. The description adds useful behavioral context about output_path persisting full data and slimming the response, but does not disclose other behaviors like rate limits or auth needs.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise with two sentences, no wasted words. It front-loads the main purpose, but could be better structured by separating the core function from the output_path detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (12 parameters) and presence of an output schema, the description covers the key behavior of output_path but lacks context on when to use different extraction types or how they differ. This is adequate but not comprehensive.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents parameters thoroughly. The description adds minimal value beyond restating extraction types and output_path behavior, meeting the baseline expectation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states the tool extracts structured data using CSS selectors or LLM, which is specific and clear. It distinguishes from sibling crawling tools but does not explicitly differentiate from other extraction tools like intelligent_extract or extract_entities.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus alternatives. It mentions the output_path behavior but lacks prerequisites or when-not-to-use instructions, which is a gap given the large set of sibling extraction tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_youtube_commentsBRead-only
Extract YouTube video comments. Supports pagination via comment_offset. Use output_path to persist the full unsliced comment list to disk as JSON; the response is then slimmed to metadata only.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | YouTube video URL | |
| sort_by | No | 'popular'|'recent' | popular |
| max_comments | No | Max comments to retrieve (1-1000) | |
| comment_offset | No | Number of comments to skip (for pagination) | |
| include_replies | No | Include reply comments | |
| content_offset | No | Start position for content (0-indexed) | |
| content_limit | No | Max characters to return (0=unlimited) | |
| output_path | No | Absolute file path (auto .json extension) to persist the full unsliced/untruncated comment list. When set, the complete comment array is written to disk BEFORE the internal token-limit reduction, and the response is slimmed (extracted_data.comments removed but comment_count etc. kept). | |
| include_content_in_response | No | When True (with output_path set), keep the comment list in the response too. Note: the response copy is still subject to content_limit/content_offset slicing and the token-limit comment-array reduction; only the on-disk file holds the full list. Defaults to False. | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Description describes a side effect (writing comments to disk) that contradicts the readOnlyHint annotation, which implies no mutations. This is a serious inconsistency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences; first states core function, second explains key persistence feature. No superfluous information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers main behavior, pagination, and file persistence, but the annotation contradiction undermines completeness. Output schema exists to cover return values.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Input schema has 100% description coverage, so baseline 3. Description adds marginal value beyond schema, clarifying output_path behavior but not essential.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states verb 'Extract' and resource 'YouTube video comments'. Differentiates from sibling tools like extract_youtube_transcript.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Implies usage for extracting comments with pagination and optional file persistence, but lacks explicit when-to-use or alternative comparisons.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extract_youtube_transcriptARead-only
Extract YouTube transcripts with timestamps. Works with public captioned videos. Supports fallback to page crawl. Use output_path to persist the full unsliced transcript to disk as markdown.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | YouTube video URL | |
| languages | No | Language codes in preference order | |
| translate_to | No | Target language for translation | |
| include_timestamps | No | Include timestamps | |
| preserve_formatting | No | Preserve formatting | |
| include_metadata | No | Include video metadata | |
| auto_summarize | No | Auto-summarize large content | |
| max_content_tokens | No | Max tokens before summarization | |
| summary_length | No | 'short'|'medium'|'long' | medium |
| llm_provider | No | LLM provider | |
| llm_model | No | LLM model | |
| enable_crawl_fallback | No | Enable page crawl fallback when API fails | |
| fallback_timeout | No | Fallback crawl timeout in seconds | |
| enrich_metadata | No | Enrich metadata (upload_date, view_count) via page crawl | |
| content_offset | No | Start position for content (0-indexed) | |
| content_limit | No | Max characters to return (0=unlimited) | |
| output_path | No | Absolute file path (auto .md extension) to persist the full unsliced transcript. When set, the response is slimmed to metadata+file path. content_limit/content_offset still affect the response copy but not the on-disk file. | |
| include_content_in_response | No | When True (with output_path set), keep the transcript in the response too. Note: the response copy is still subject to content_limit/content_offset slicing; only the on-disk file holds the full transcript. Defaults to False. | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description adds value beyond annotations (readOnlyHint: true) by disclosing fallback crawling behavior, output_path persistence, metadata enrichment, and auto-summarization. No contradictions found. Could mention rate limits or video length constraints.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four sentences, front-loaded with the core purpose. Each sentence adds distinct information: extraction, constraints, fallback, output guidance. Slightly verbose but no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (19 params, output schema exists), the description covers key behaviors: extraction, fallback, output persistence, summarization, metadata enrichment. It omits rate limits and language constraints but the schema covers languages. Output schema reduces need to describe return values.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
All 19 parameters have schema descriptions (100% coverage), and the description adds context for output_path (persist full unsliced transcript as markdown) and fallback. The description does not override schema but complements it effectively.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool extracts YouTube transcripts with timestamps, specifying the resource (YouTube) and action (extract). It distinguishes from siblings like get_youtube_video_info (video metadata) and batch_extract_youtube_transcripts (batch variant).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides context: works only with public captioned videos, supports fallback to page crawl, and suggests using output_path for persistence. However, it does not explicitly compare to sibling tools like extract_youtube_comments or state when to avoid using this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_search_genresARead-only
Get available search genres for targeted searching.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and openWorldHint=false. The description adds no further behavioral details such as whether the genre list is static or dynamic, or any rate limits. It is minimally adequate given the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence with the verb 'Get' front-loaded, containing no unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple tool with no parameters and an output schema, the description sufficiently conveys the tool's purpose. The agent can rely on the output schema for return structure details.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has zero parameters, so the description does not need to explain them. The baseline score of 4 is appropriate as the description does not contradict or fail to add value.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves available search genres for targeted searching, using a specific verb ('Get') and resource ('search genres'). It is distinct from sibling tools that perform searches themselves.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when needing genre options for a targeted search, but does not explicitly state when not to use it or provide alternatives among siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_supported_file_formatsARead-only
Get supported file formats (PDF, Office, ZIP) and their capabilities.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations indicate readOnlyHint=true. The description adds context by specifying it returns format capabilities, which goes beyond the annotation. No contradictions.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence that front-loads the action and examples. No extraneous words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given zero parameters and the presence of an output schema, the description sufficiently informs the agent about the tool's purpose and return value.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are no parameters, so the description does not need to add meaning beyond the input schema. The schema is fully covered.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves supported file formats and their capabilities, with specific examples (PDF, Office, ZIP). It is distinct from sibling tools which focus on crawling, extraction, or processing.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The context is clear: this tool is used to check which formats are supported before using other file-processing tools. No explicit exclusions or alternatives are needed given the tool's simplicity.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_youtube_video_infoARead-only
Get YouTube video metadata and transcript availability. Use output_path to persist the full transcript to disk as markdown and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| video_url | Yes | YouTube video URL | |
| summarize_transcript | No | Summarize transcript | |
| max_tokens | No | Token limit for summarization | |
| llm_provider | No | LLM provider | |
| llm_model | No | LLM model | |
| summary_length | No | 'short'|'medium'|'long' | medium |
| include_timestamps | No | Include timestamps | |
| output_path | No | Absolute file path (auto .md extension) to persist the full video info + transcript as markdown. When set, the response is slimmed to metadata+file path. | |
| include_content_in_response | No | When True (with output_path set), also include the full transcript in the response. Defaults to False. | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare readOnlyHint=true. The description adds that persisting to disk writes to a local file and alters response format. This provides behavioral context beyond the annotation, though the side effect of file writing is still safe.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two concise sentences, front-loaded with purpose and key option. No unnecessary words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Description covers core purpose and one key parameter, but with 10 parameters and sibling tools, more context on summarization and differentiation would be beneficial. Output schema covers return values, so not necessary in description.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so parameters are well-documented. The description adds value for output_path by explaining its effect on response, but other parameters like summarize_transcript are not elaborated beyond schema defaults.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves video metadata and transcript availability, and mentions persisting transcript to disk. Verb 'Get' is specific. However, it does not explicitly differentiate from sibling tools like extract_youtube_transcript.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description hints at using output_path but provides no guidance on when to use this tool versus alternatives like extract_youtube_transcript. No context on exclusions or prerequisites.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
intelligent_extractARead-only
Extract specific data from web pages using LLM. Use output_path to persist the full extraction output to disk as JSON and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Target URL | |
| extraction_goal | Yes | Data to extract | |
| content_filter | No | 'bm25'|'pruning'|'llm' | bm25 |
| filter_query | No | BM25 filter keywords | |
| chunk_content | No | Split content | |
| use_llm | No | Enable LLM | |
| llm_provider | No | LLM provider | |
| llm_model | No | LLM model | |
| custom_instructions | No | LLM instructions | |
| output_path | No | Absolute file path (auto .json extension) to persist the full extracted data + content as JSON. When set, the response is slimmed to metadata+file path (extracted_data.raw_content, content, markdown, table_data removed). | |
| include_content_in_response | No | When True (with output_path set), also keep extracted_data/content in the response. Defaults to False. | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations indicate readOnlyHint=true, so the tool is non-destructive. The description adds the behavior of output_path enabling slim responses. No further details on rate limits, error handling, or LLM behavior are provided. Does not contradict annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences: first states purpose, second explains key workflow parameter. No extraneous information. Efficient and front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (12 parameters, output schema present), the description covers the main purpose and the critical output_path behavior. It could be improved by clarifying when to use this over crawling or batch tools, but overall it is sufficient for an agent to understand the tool's role.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so most parameter meanings are already clear. The description adds context for output_path, explaining its effect on response structure. This provides marginal additional value beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool extracts specific data from web pages using LLM, distinguishing it from sibling tools like crawl_url which likely return full content. The verb 'extract' and resource 'web pages' are specific.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for extracting data from web pages with optional output persistence, but does not explicitly state when to use this tool versus alternatives like batch_crawl or search_and_crawl. No exclusions or prerequisites are mentioned.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
multi_url_crawlARead-only
Multi-URL crawl with pattern-based config. Max 5 URL patterns per call. Use output_path (directory) to persist full per-URL markdown + index.json; the return shape stays a list, each success item gets an output_file key.
| Name | Required | Description | Default |
|---|---|---|---|
| url_configurations | Yes | URL-config mapping (max 5 URLs). Example: {'https://site1.com': {'wait_for_js': true}} | |
| pattern_matching | No | Pattern: 'wildcard' or 'regex' (default: wildcard) | wildcard |
| default_config | No | Default config | |
| base_timeout | No | Timeout per URL (default: 30) | |
| max_concurrent | No | Max concurrent (default: 3) | |
| output_path | No | Absolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names are fine). The list return shape is preserved; each successful item gains an 'output_file' key. Failed items (success=False) are NOT written as .md but still appear in index.json with file=null. | |
| include_content_in_response | No | When True (with output_path), keep full markdown/content in each list item. Defaults to False. | |
| overwrite | No | Overwrite existing per-URL files inside output_path. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description details output_path behavior: persisting per-URL markdown + index.json, return shape preservation, output_file key, and handling of failed items. This adds significant value beyond annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences with no wasted words, front-loaded with action and key constraints. Highly efficient.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given complexity (8 parameters, nested objects, output schema), the description covers core behavior, output path details, and return shape. Output schema exists, so return values are explained elsewhere.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, but the description adds constraints like max 5 URLs and clarifies return shape, which are not in schema. This provides meaningful extra context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it performs multi-URL crawls with pattern-based config, distinguishing it from single-URL tools like crawl_url. It specifies max 5 URL patterns and mentions output persistence, which is unique.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies use for multiple URLs with pattern matching and mentions max 5 patterns. It does not explicitly exclude other tools but provides enough context for selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
process_fileBRead-only
Convert PDF, Word, Excel, PowerPoint, ZIP to markdown. Use output_path to persist the full unsliced converted markdown to disk and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | File URL or local path (PDF, Office, ZIP). Supports http/https URLs, file:// URIs, and absolute paths. | |
| max_size_mb | No | Max file size in MB | |
| extract_all_from_zip | No | Extract ZIP contents | |
| include_metadata | No | Include metadata | |
| auto_summarize | No | Auto-summarize large content | |
| max_content_tokens | No | Max tokens before summarization | |
| summary_length | No | 'short'|'medium'|'long' | medium |
| llm_provider | No | LLM provider | |
| llm_model | No | LLM model | |
| content_limit | No | Max characters to return (0=unlimited) | |
| content_offset | No | Start position for content (0-indexed) | |
| output_path | No | Absolute file path (auto .md extension) to persist the full unsliced converted markdown. When set, the response is slimmed to metadata+file path. content_limit/content_offset still affect the response copy but not the on-disk file. | |
| include_content_in_response | No | When True (with output_path set), keep content in the response too. Note: the response copy is still subject to content_limit/content_offset slicing; only the on-disk file holds the full unsliced payload. Defaults to False. | |
| overwrite | No | Overwrite an existing output file at output_path. Defaults to False. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description states the tool 'converts' files, implying a write/mutation operation, but the annotation declares readOnlyHint=true. This is a direct contradiction and severely misleads the agent about the tool's side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences: first states core purpose, second explains key usage pattern. Every word is necessary and front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Essentials covered: conversion types and output_path. With full schema coverage and output schema present, the description is sufficiently complete for a complex tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, baseline 3. The description adds value by explaining the output_path behavior (slim response, full unsliced disk persistence) and slicing effect, going beyond schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool converts PDF, Word, Excel, PowerPoint, ZIP to markdown, and mentions the key output_path feature. It is specific and informative.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus siblings like batch_crawl or crawl_url. The description does not mention alternatives or exclusions.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_and_crawlARead-only
Search Google and crawl top results. Combines search with full content extraction. Supply output_path (directory) in the request to persist per-page markdown (unsliced) + index.json and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| request | Yes | Dict with: search_query (required), crawl_top_results, search_genre, recent_days, generate_markdown, max_content_per_page. Optional persistence keys: output_path (absolute directory — per-page .md files + index.json, the full page bodies are written BEFORE max_content_per_page truncation; dot-containing dir names are fine), include_content_in_response (bool, default False — when True keeps crawled_pages in the response too, still subject to max_content_per_page truncation), overwrite (bool, default False). Failed pages appear in index.json with file=null. |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations set readOnlyHint=true, but the description describes writing files to disk (per-page markdown, index.json), which is a side effect. This is a direct contradiction. The description does not acknowledge or clarify the discrepancy, nor does it explain any non-destructive nature.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is succinct with only two sentences, yet covers the core purpose, output mechanism, and key configurations. Every sentence adds essential information; no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite the complexity of combining search and crawl with many options, the description covers input, persistence, response behavior, and error handling. However, it omits details on rate limiting, timeouts, or the output schema structure, which would enhance completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The parameter description inside the input schema is highly detailed, explaining each key's purpose, default behavior (e.g., 'full page bodies are written BEFORE truncation'), and special cases (failed pages). This adds significant value beyond the schema structure, achieving high clarity.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool combines Google search with crawling top results, specifying verbs 'Search' and 'crawl' on the resource 'Google results'. It distinctively differentiates from siblings like 'search_google' and 'crawl_url' by highlighting the combination.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides instructions on when to use (to combine search and crawl with persistence) but lacks explicit guidance on when not to use it or alternatives. No mention of siblings or trade-offs.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
search_googleARead-only
Search Google with genre filtering. Genres: academic, news, technical, commercial, social. Supply output_path in the request to persist the full unsliced result set to disk as JSON and receive a slim response.
| Name | Required | Description | Default |
|---|---|---|---|
| request | Yes | Dict with: query (required), num_results, search_genre, language, region, recent_days, content_limit (int), content_offset (int). Optional persistence keys: output_path (absolute file path, auto .json extension — full unsliced results written to disk BEFORE content_limit/content_offset slicing), include_content_in_response (bool, default False — when True keeps results in the response too, still subject to slicing), overwrite (bool, default False). |
Output Schema
| Name | Required | Description |
|---|---|---|
No output parameters | ||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations declare readOnlyHint:true. Description adds optional disk persistence via output_path and slim response behavior, providing useful context beyond annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, front-loaded with purpose and genres, and a second sentence for persistence. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the existence of an output schema, the description adequately covers genre filtering and optional persistence. It omits details on result slicing but the schema covers that.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100% with detailed descriptions for all sub-parameters. The tool description reinforces the output_path key but adds no new meaning beyond existing schema documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Search Google with genre filtering', listing specific genres. This distinguishes it from siblings like batch_search_google or search_and_crawl.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies use for single Google searches with genre filtering but does not explicitly compare to sibling tools or state when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
19 tool updates
v0.3.2- Added
batch_crawl - Added
batch_extract_youtube_transcripts - Added
batch_search_google - Added
crawl_url - Added
crawl_url_with_fallback - Added
deep_crawl_site - Added
enhanced_process_large_content - Added
extract_entities - Added
extract_structured_data - Added
extract_youtube_comments - Added
extract_youtube_transcript - Added
get_search_genres - Added
get_supported_file_formats - Added
get_youtube_video_info - Added
intelligent_extract - Added
multi_url_crawl - Added
process_file - Added
search_and_crawl - Added
search_google
13 tool updates
v0.1.6- Removed
batch_crawl - Removed
batch_extract_youtube_transcripts - Removed
batch_search_google - Removed
crawl_url - Removed
crawl_url_with_fallback - Removed
deep_crawl_site - Removed
extract_structured_data - Removed
extract_youtube_transcript - Removed
get_youtube_video_info - Removed
multi_url_crawl - Removed
process_file - Removed
search_and_crawl - Removed
search_google
13 tool updates
v0.1.7- First observed
batch_crawl - First observed
batch_extract_youtube_transcripts - First observed
batch_search_google - First observed
crawl_url - First observed
crawl_url_with_fallback - First observed
deep_crawl_site - First observed
extract_structured_data - First observed
extract_youtube_transcript - First observed
get_youtube_video_info - First observed
multi_url_crawl - First observed
process_file - First observed
search_and_crawl - First observed
search_google
TDQS
Scored across 19 tools
Most tools have distinct purposes, but there is potential overlap between batch_crawl and multi_url_crawl, and between single/batch versions of search and YouTube extraction. Descriptions help disambiguate, but slight confusion may occur.
All tools follow a consistent verb_noun snake_case pattern, with batch_ prefix for batch operations and get_ prefix for metadata retrieval. No mixing of conventions.
19 tools cover a broad domain (web crawling, search, YouTube, file processing, extraction) without being excessive. Each tool has a clear role and the count feels well-scoped.
The tool set covers the full lifecycle of web data extraction: crawling (single, batch, deep), search, YouTube operations, file conversion, and multiple extraction methods (entities, structured, intelligent). No obvious gaps for the stated purpose.
Maintenance
Related MCP Connectors
MCP server for Firecrawl — web search, scraping, and biomedical/arXiv paper search.
An MCP server that gives your AI access to the source code and docs of all public github repos
MCP server for OpenAI API (chat completions, image generation, embeddings) via AceDataCloud
Turn any public website into an MCP server for agents to search, read and navigate.
Related MCP Servers
- FlicenseBqualityDmaintenanceAn MCP Server for Web scraping and Crawling, built using Crawl4AI224-
- FlicenseNot gradedqualityDmaintenanceA locally-hosted MCP server that provides AI assistants with advanced web crawling capabilities, including structured data extraction, deep site crawling, and page screenshots. It enables users to convert single or multiple URLs into clean Markdown content for processing by LLMs without requiring external API keys for basic features.-
- AlicenseAqualityBmaintenanceWeb extraction MCP server for AI agents. Extract structured data from any URL with built-in Cloudflare bypass, JavaScript rendering, and intelligent parsing. Returns clean markdown or JSON.5794 npm2MIT
- AlicenseNot gradedqualityCmaintenanceMCP server integrating Crawl4AI for universal web crawling and data extraction. Enables AI agents to crawl, extract markdown/HTML, take screenshots, generate PDFs, and execute JavaScript on web pages.23 npm7MIT