Skip to main content
Glama

Crawl-MCP: Unofficial MCP Server for crawl4ai

⚠️ Important: This is an unofficial MCP server implementation for the excellent crawl4ai library.
Not affiliated with the original crawl4ai project.

A comprehensive Model Context Protocol (MCP) server that wraps the powerful crawl4ai library with advanced AI capabilities. Extract and analyze content from any source: web pages, PDFs, Office documents, YouTube videos, and more. Features intelligent summarization to dramatically reduce token usage while preserving key information.

🌟 Key Features

  • 🔍 Google Search Integration - 7 optimized search genres with Google official operators

  • 🔍 Advanced Web Crawling: JavaScript support, deep site mapping, entity extraction

  • 🌐 Universal Content Extraction: Web pages, PDFs, Word docs, Excel, PowerPoint, ZIP archives

  • 🤖 AI-Powered Summarization: Smart token reduction (up to 88.5%) while preserving essential information

  • 💾 Disk Persistence (token-saver): Save full results to disk and return slim metadata, so agents read them from a file on demand without spending context

  • 🎬 YouTube Integration: Extract video transcripts and summaries without API keys

  • ⚡ Production Ready: 19 specialized tools with comprehensive error handling

Related MCP server: Crawl4AI MCP Server

🚀 Quick Start

Prerequisites (Required First)

  • Python 3.11 or later (FastMCP requires Python 3.11+)

Install system dependencies for Playwright:

Ubuntu 24.04 LTS (Manual Required):

# Manual setup required due to t64 library transition
sudo apt update && sudo apt install -y \
  libnss3 libatk-bridge2.0-0 libxss1 libasound2t64 \
  libgbm1 libgtk-3-0t64 libxshmfence-dev libxrandr2 \
  libxcomposite1 libxcursor1 libxdamage1 libxi6 \
  fonts-noto-color-emoji fonts-unifont python3-venv python3-pip

python3 -m venv venv && source venv/bin/activate
pip install playwright==1.55.0 && playwright install chromium
sudo playwright install-deps

Other Linux/macOS:

sudo bash scripts/prepare_for_uvx_playwright.sh

Windows (as Administrator):

scripts/prepare_for_uvx_playwright.ps1

Installation

UVX (Recommended - Easiest):

# After system preparation above - that's it!
uvx --from git+https://github.com/walksoda/crawl-mcp crawl-mcp

Docker (Production-Ready):

# Clone the repository
git clone https://github.com/walksoda/crawl-mcp
cd crawl-mcp

# Build and run with Docker Compose (STDIO mode)
docker-compose up --build

# Or build and run HTTP mode on port 8000
docker-compose --profile http up --build crawl4ai-mcp-http

# Or build manually
docker build -t crawl4ai-mcp .
docker run -it crawl4ai-mcp

Docker Features:

  • 🔧 Multi-Browser Support: Chromium, Firefox, Webkit headless browsers

  • 🐧 Google Chrome: Additional Chrome Stable for compatibility

  • Optimized Performance: Pre-configured browser flags for Docker

  • 🔒 Security: Non-root user execution

  • 📦 Complete Dependencies: All required libraries included

Claude Desktop Setup

UVX Installation: Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "crawl-mcp": {
      "transport": "stdio",
      "command": "uvx",
      "args": [
        "--from",
        "git+https://github.com/walksoda/crawl-mcp",
        "crawl-mcp"
      ],
      "env": {
        "CRAWL4AI_LANG": "en"
      }
    }
  }
}

Docker HTTP Mode:

{
  "mcpServers": {
    "crawl-mcp": {
      "transport": "http",
      "baseUrl": "http://localhost:8000"
    }
  }
}

For Japanese interface:

"env": {
  "CRAWL4AI_LANG": "ja"
}

📖 Documentation

Topic

Description

Installation Guide

Complete installation instructions for all platforms

API Reference

Full tool documentation and usage examples

Configuration Examples

Platform-specific setup configurations

HTTP Integration

HTTP API access and integration methods

Advanced Usage

Power user techniques and workflows

Development Guide

Contributing and development setup

Language-Specific Documentation

🛠️ Tool Overview

Web Crawling (3)

  • crawl_url - Extract web page content with JavaScript support

  • deep_crawl_site - Crawl multiple pages from a site with configurable depth

  • crawl_url_with_fallback - Crawl with fallback strategies for anti-bot sites

Data Extraction (3)

  • intelligent_extract - Extract specific data from web pages using LLM

  • extract_entities - Extract entities (emails, phones, etc.) from web pages

  • extract_structured_data - Extract structured data using CSS selectors or LLM

YouTube (4)

  • extract_youtube_transcript - Extract YouTube transcripts with timestamps

  • batch_extract_youtube_transcripts - Extract transcripts from multiple YouTube videos (max 3)

  • get_youtube_video_info - Get YouTube video metadata and transcript availability

  • extract_youtube_comments - Extract YouTube video comments with pagination

Search (4)

  • search_google - Search Google with genre filtering

  • batch_search_google - Perform multiple Google searches (max 3)

  • search_and_crawl - Search Google and crawl top results

  • get_search_genres - Get available search genres

File Processing (3)

  • process_file - Convert PDF, Word, Excel, PowerPoint, ZIP to markdown

  • get_supported_file_formats - Get supported file formats and capabilities

  • enhanced_process_large_content - Process large content with chunking and BM25 filtering

Batch Operations (2)

  • batch_crawl - Crawl multiple URLs with fallback (max 3 URLs)

  • multi_url_crawl - Multi-URL crawl with pattern-based config (max 5 URL patterns)

💾 Persist Large Results to Disk (token-saver)

All information-gathering tools accept an optional output_path parameter that writes the full fetched content straight to disk and returns a slim metadata-only response. This lets an LLM fetch huge pages, long YouTube transcripts, or whole batches without blowing its context budget — read from the saved file only when needed.

How it works:

  • Single-file tools (e.g. crawl_url, extract_youtube_transcript) write one .md (or .json for JSON-kind tools) — pass an absolute file path; the extension is auto-added if omitted. An existing regular file at that path is rejected unless overwrite=true.

  • Batch tools (batch_crawl, multi_url_crawl, deep_crawl_site, search_and_crawl, batch_extract_youtube_transcripts) expect an absolute directory path and write one .md per URL plus index.json. Any non-existent path is treated as a directory and created — including names containing dots such as /tmp/run.v1. If the path already exists as a regular file, the call is rejected. batch_crawl / multi_url_crawl keep their list return shape and embed an output_file key on each success item.

  • Request-dict tools (search_google, batch_search_google, search_and_crawl, batch_extract_youtube_transcripts) read the persistence keys directly from their request dict.

  • Common parameters: output_path (absolute; None or "" skips persistence), include_content_in_response (default false — when true, content is included in the response too, still subject to any content_limit/content_offset/max_content_per_page slicing), overwrite (default false).

  • Writes are atomic per file (temp file + os.replace); parent directories are auto-created; the full unsliced payload is persisted before any slicing or tool-internal truncation so the on-disk copy is always complete even when the response is sliced.

  • Batch dict tools (deep_crawl_site, search_and_crawl, batch_extract_youtube_transcripts) skip per-item persistence for items that report success=false; these still appear in index.json with file: null so callers can reason about the attempt list.

Markdown single-file example:

{
  "tool": "crawl_url",
  "arguments": {
    "url": "https://example.com/long-article",
    "output_path": "/tmp/crawl_out/article.md"
  }
}

JSON structured extraction (extension auto-added):

{
  "tool": "extract_structured_data",
  "arguments": {
    "url": "https://example.com/products",
    "extraction_type": "css",
    "css_selectors": {"price": ".price", "name": "h1"},
    "output_path": "/tmp/crawl_out/products"
  }
}

Batch directory mode:

{
  "tool": "batch_crawl",
  "arguments": {
    "urls": ["https://a.example", "https://b.example"],
    "output_path": "/tmp/crawl_out/batch_run1"
  }
}

Each persisted markdown file begins with a YAML frontmatter block containing url, title, fetched_at, and source_tool so the artifact is self-describing.

🎯 Common Use Cases

Content Research:

search_and_crawl → extract_structured_data → analysis

Documentation Mining:

deep_crawl_site → batch processing → extraction

Media Analysis:

extract_youtube_transcript → summarization workflow

Site Mapping:

batch_crawl → multi_url_crawl → comprehensive data

🚨 Quick Troubleshooting

Installation Issues:

  1. Re-run setup scripts with proper privileges

  2. Try development installation method

  3. Check browser dependencies are installed

Performance Issues:

  • Use wait_for_js: true for JavaScript-heavy sites

  • Increase timeout for slow-loading pages

  • Use extract_structured_data for targeted extraction

Configuration Issues:

  • Check JSON syntax in claude_desktop_config.json

  • Verify file paths are absolute

  • Restart Claude Desktop after configuration changes

🏗️ Project Structure

  • Original Library: crawl4ai by unclecode

  • MCP Wrapper: This repository (walksoda)

  • Implementation: Unofficial third-party integration

📄 License

This project is an unofficial wrapper around the crawl4ai library. Please refer to the original crawl4ai license for the underlying functionality.

🤝 Contributing

See our Development Guide for contribution guidelines and development setup instructions.

Available Tools

19 tools
batch_crawlA

Crawl multiple URLs with fallback. Max 3 URLs per call. Use output_path (directory) to persist full per-URL markdown + index.json; the return shape stays a list, each success item gets an output_file key.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlsYesURLs to crawl (max 3)
base_timeoutNoTimeout per URL (default: 30)
generate_markdownNoGenerate markdown (default: True)
extract_mediaNoExtract media (default: False)
wait_for_jsNoWait for JS (default: False)
output_pathNoAbsolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names like /tmp/run.v1 are fine). The list return shape is preserved; each successful item gains an 'output_file' key. Failed items (success=False) are NOT written as .md but still appear in index.json with file=null.
include_content_in_responseNoWhen True (with output_path), keep full markdown/content in each list item. Defaults to False so the response stays token-efficient.
overwriteNoOverwrite existing per-URL files inside output_path. Defaults to False (existing files cause an output_path_exists error, returned as a single-element list).

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Without annotations, the description covers key behaviors: output_path creation and validation, per-URL file writing, error handling for failures and existing files, and the effect of include_content_in_response. It does not mention authentication or rate limits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, compact paragraph that front-loads the core purpose and constraints. It packs significant detail without redundancy, though it could benefit from slight restructuring for readability.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (8 parameters, output schema present), the description covers most behavioral aspects and parameter interactions. A minor gap is the lack of elaboration on the fallback mechanism mentioned in the title.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, but the description adds meaningful context for output_path (e.g., dot-containing names allowed, index.json handling) and include_content_in_response (token efficiency). This goes beyond the schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool crawls multiple URLs with fallback and specifies a maximum of 3 URLs per call. It identifies the resource and action, but does not explicitly differentiate from siblings like multi_url_crawl or crawl_url_with_fallback.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides guidance on using output_path for persistence and notes constraints like maximum URLs. However, it lacks explicit context on when to choose this tool over alternatives or when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

batch_extract_youtube_transcriptsB
Read-only

Extract transcripts from multiple YouTube videos. Max 3 URLs per call. Supply output_path (directory) in the request to persist per-video markdown files + index.json and receive a slim response.

ParametersJSON Schema
NameRequiredDescriptionDefault
requestYesDict with: urls (max 3), languages, include_timestamps. Optional persistence keys: output_path (absolute directory — per-video .md files + index.json; dot-containing dir names are fine), include_content_in_response (bool; default False — when True, per-video transcripts stay in the response as well), overwrite (bool; default False — existing files rejected). Failed items (success=False) are recorded in index.json with file=null but no .md is written.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.4/5.0
Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotation declares readOnlyHint=true, implying no state changes, but description and schema describe writing .md files and index.json to disk, which modifies the filesystem. This contradiction is severe, and per rubric, score is 1.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences front-load the purpose and state key constraints (max URLs, output_path requirement) without extraneous information. Every word is necessary and no fluff.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite the annotation contradiction, the description covers purpose, capacity, and persistence requirement. The schema fills in parameter details, and output schema exists. Missing some behavioral details like error handling for failed items (mentioned only in schema), but overall sufficient for invocation given the rich schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, and the parameter description in the schema already details all keys (urls, languages, output_path, etc.). The tool description adds minimal value beyond what the schema provides, so baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states the tool extracts transcripts from multiple YouTube videos, distinguishing it from the singular counterpart extract_youtube_transcript. It specifies a max of 3 URLs per call and mentions persistence, which is specific and actionable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Description provides constraints (max 3 URLs, need output_path) but does not explicitly guide when to use this batch tool versus the singular extract_youtube_transcript or other sibling tools. Usage context is implied but exclusions and alternatives are not stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

batch_search_googleA
Read-only

Perform multiple Google searches. Max 3 queries per call. Supply output_path in the request to persist the full result set to disk as JSON and receive a slim response.

ParametersJSON Schema
NameRequiredDescriptionDefault
requestYesDict with: queries (max 3), num_results_per_query, search_genre, recent_days. Optional persistence keys: output_path (absolute file path, auto .json extension — full result set written to disk), include_content_in_response (bool, default False — when True keeps results in the response too), overwrite (bool, default False).

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations provide readOnlyHint=true, consistent with search. Description adds persistence behavior (writing to disk) and slim response, but lacks details on error handling or quota implications. No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences pack the core action, limit, and optional persistence, with no redundancy. Front-loaded with purpose and constraint.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given rich schema descriptions and output schema existence, the description adequately conveys the core function. Missing behavioral details like error responses or handling of partial failures, but sufficient for typical use.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, with detailed parameter descriptions. The main description adds minimal extra value beyond schema, achieving baseline for a fully covered schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description explicitly states it performs multiple Google searches with a max of 3 queries, clearly distinguishing it from single-query tools like search_google and crawling tools like batch_crawl.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description indicates when to use (multiple searches) but lacks explicit when-not-to-use or sibling comparisons. It implicitly suggests use for batch queries over single search but could be more direct.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_urlA

Extract web page content with JavaScript support. Use wait_for_js=true for SPAs. Use content_offset/content_limit to paginate the response. Use output_path to persist the full unsliced content to disk as markdown and receive a slim metadata-only response.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesURL to crawl
css_selectorNoCSS selector for extraction
extract_mediaNoExtract images/videos
take_screenshotNoTake screenshot
generate_markdownNoGenerate markdown
include_cleaned_htmlNoInclude cleaned HTML
wait_for_selectorNoWait for element to load
timeoutNoTimeout in seconds
wait_for_jsNoWait for JavaScript
auto_summarizeNoAuto-summarize large content
use_undetected_browserNoBypass bot detection
content_limitNoMax characters to return (0=unlimited)
content_offsetNoStart position for content (0-indexed)
output_pathNoAbsolute file path (auto .md extension) to persist the full unsliced markdown. When set, the response is slimmed to metadata+file path to save tokens. content_limit/content_offset still affect the response copy but not the on-disk file.
include_content_in_responseNoWhen True (with output_path set), keep markdown/content in the response too. Note: the response copy is still subject to content_limit/content_offset slicing; only the on-disk file holds the full unsliced payload. Defaults to False.
overwriteNoOverwrite an existing output file at output_path. Defaults to False (existing files are rejected before any fetch).

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description adds significant behavioral detail: how output_path slimdown works, the interaction between content_offset/limit and the on-disk file, and the include_content_in_response flag. This goes beyond basic functionality.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise: three short sentences that front-load the purpose and quickly cover the most important usage scenarios. No unnecessary words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 16 parameters and an output schema, the description covers the most important behaviors (JS, pagination, persistence) but omits details on less common params like css_selector or extract_media. Still, the core agent needs are addressed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description adds value by explaining the interplay between output_path, content_offset, and content_limit, and advises when to use wait_for_js, which is not present in the schema alone.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool extracts web page content with JavaScript support, and provides specific use cases like SPAs and pagination. However, it does not differentiate from sibling tools like 'crawl_url_with_fallback' or 'deep_crawl_site', which lowers the score slightly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit usage guidelines for key features (wait_for_js for SPAs, pagination with offsets, persisting content to disk). It lacks guidance on when not to use this tool or alternatives, but provides enough context for common scenarios.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

crawl_url_with_fallbackA
Read-only

Crawl with fallback strategies for anti-bot sites. Use content_offset/content_limit to paginate the response. Use output_path to persist the full unsliced content to disk as markdown and receive a slim response.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesURL to crawl
css_selectorNoCSS selector
extract_mediaNoExtract media
take_screenshotNoTake screenshot
generate_markdownNoGenerate markdown
wait_for_selectorNoElement to wait for
timeoutNoTimeout in seconds
wait_for_jsNoWait for JavaScript
auto_summarizeNoAuto-summarize content
content_limitNoMax characters to return (0=unlimited)
content_offsetNoStart position for content (0-indexed)
output_pathNoAbsolute file path (auto .md extension) to persist the full unsliced markdown. When set, the response is slimmed to metadata+file path. content_limit/content_offset still affect the response copy but not the on-disk file.
include_content_in_responseNoWhen True (with output_path set), keep markdown/content in the response too. Note: the response copy is still subject to content_limit/content_offset slicing; only the on-disk file holds the full unsliced payload.
overwriteNoOverwrite an existing output file at output_path. Defaults to False (existing files rejected before any fetch).

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate readOnlyHint=true, which is consistent with crawling. Description adds that fallback strategies exist for anti-bot sites, but does not detail what those strategies are or any other behavioral traits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with core purpose, no fluff. Each sentence adds distinct value: purpose, pagination, and persistence.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given 14 parameters and an output schema (not shown), the description covers the main behavioral aspects (fallback, pagination, persistence). Minor gap: no explanation of the fallback strategies themselves, but overall adequate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 100% schema coverage, baseline is 3. The description adds meaningful context for pagination (content_offset/content_limit) and persistence (output_path with behavior rules), going beyond the schema comments.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's action ('Crawl') and target ('anti-bot sites') with fallback strategies, but does not differentiate from the sibling tool 'crawl_url' which likely has similar purpose.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage via mention of pagination and persistence, but does not provide explicit guidance on when to choose this tool over alternatives like 'crawl_url' or 'search_and_crawl'.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

deep_crawl_siteA
Read-only

Crawl multiple pages from a site with configurable depth. Use output_path (directory) to persist per-URL markdown files + index.json; the response is then slimmed to metadata only.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesStarting URL
max_depthNoLink depth (1-2)
max_pagesNoMax pages (max: 10)
crawl_strategyNo'bfs'|'dfs'|'best_first'bfs
include_externalNoFollow external links
url_patternNoURL filter pattern
score_thresholdNoMin relevance 0-1
extract_mediaNoExtract media
base_timeoutNoTimeout per page
output_pathNoAbsolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names like /tmp/run.v1 are fine). When set, the response is slimmed to metadata+file paths. Failed items (success=False) are NOT written as .md but still recorded in index.json with file=null.
include_content_in_responseNoWhen True (with output_path set), also include per-page content/markdown in the response items. Defaults to False so the response stays token-efficient.
overwriteNoOverwrite existing per-URL files inside output_path. Defaults to False (existing files cause an output_path_exists error).

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description discloses that writing files to output_path is a side effect, and that the response is slimmed when output_path is set. This adds value beyond the readOnlyHint annotation, but it does not cover all behavioral traits (e.g., rate limits, auth).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with purpose and key behavior. Every sentence earns its place; no redundancy with schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given tool complexity (12 parameters) and presence of output schema, the description is fairly complete. It explains the main output_path feature and behavior, but does not elaborate on other parameters or when to use depth vs alternatives.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description adds significant meaning to parameters like output_path, include_content_in_response, and overwrite, explaining their effects on response and file persistence. Schema coverage is 100% but the description enriches understanding.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it crawls multiple pages from a site with configurable depth, using a verb+resource structure. It distinguishes from siblings like crawl_url by mentioning multi-page and output persistence.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for multi-page crawling with depth, but does not explicitly state when to use this tool versus alternatives like crawl_url or batch_crawl. No when-not or exclusions are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

enhanced_process_large_contentB
Read-only

Process large content with chunking and BM25 filtering. Use output_path to persist chunks + summaries to disk as JSON and receive a slim response.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesURL to process
chunking_strategyNo'topic'|'sentence'|'overlap'|'regex'sentence
filtering_strategyNo'bm25'|'pruning'|'llm'bm25
filter_queryNoKeywords for BM25 filtering
max_chunk_tokensNoMax tokens per chunk
chunk_overlapNoOverlap tokens
extract_top_chunksNoTop chunks to extract
similarity_thresholdNoMin similarity 0-1
summarize_chunksNoSummarize chunks
merge_strategyNo'hierarchical'|'linear'linear
final_summary_lengthNo'short'|'medium'|'long'short
output_pathNoAbsolute file path (auto .json extension) to persist the full chunks + summaries as JSON. When set, the response is slimmed to metadata+file path (chunks, chunk_summaries, merged_summary, final_summary removed).
include_content_in_responseNoWhen True (with output_path set), also include chunks/summaries in the response. Defaults to False.
overwriteNoOverwrite an existing output file at output_path. Defaults to False.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3/5.0
Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotations declare readOnlyHint=true, but the description states the tool persists data to disk, which is a write operation. This is a direct contradiction. Additionally, the description does not disclose other behavioral traits such as required permissions, rate limits, or side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences long, directly stating the core functionality and key usage advice. Every word earns its place with no fluff.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's 14 parameters and existence of an output schema, the description is minimal but covers the main point. However, it lacks details about return structure, error conditions, and behavioral nuances beyond the output_path feature.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Input schema has 100% coverage for all 14 parameters, providing clear descriptions. The description adds value by explaining the effect of output_path and the slim response, which enhances understanding beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it processes large content with chunking and BM25 filtering, and mentions persisting to disk. It distinguishes itself from siblings by specifying the advanced processing features, though it doesn't explicitly differentiate from similar tools like 'process_file'.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided on when to use this tool versus alternatives. It only implies usage via the description but lacks explicit context, exclusions, or comparisons with sibling tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_entitiesA
Read-only

Extract entities (emails, phones, etc.) from web pages. Use output_path to persist the full entity extraction output to disk as JSON and receive a slim response.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesTarget URL
entity_typesYesTypes: email, phone, url, date, ip, price
custom_patternsNoCustom regex patterns
include_contextNoInclude context
deduplicateNoRemove duplicates
use_llmNoUse LLM for NER
llm_providerNoLLM provider
llm_modelNoLLM model
output_pathNoAbsolute file path (auto .json extension) to persist the full entity extraction as JSON. When set, the response is slimmed (content, markdown, extracted_data.raw_content removed).
include_content_in_responseNoWhen True (with output_path set), also keep the entity data in the response. Defaults to False.
overwriteNoOverwrite an existing output file at output_path. Defaults to False.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Description adds context beyond readOnlyHint annotation, detailing the output_path parameter behavior (persisting full output to disk and returning slim response). No contradictions with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with purpose, no wasted words. Efficient and well-structured.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Tool has output schema covering return values, and description explains key behavioral nuance. For a complex 11-parameter tool, description is brief but adequate given schema richness.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline 3 is appropriate. Description enhances understanding of the output_path parameter, explaining its effect on response format, which exceeds baseline.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states 'Extract entities (emails, phones, etc.) from web pages,' providing a specific verb and resource with examples. This distinguishes it from siblings like extract_structured_data and intelligent_extract.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Description does not provide explicit when-to-use or when-not-to-use guidance. No mention of alternatives or context for selecting this tool over similar siblings like extract_structured_data.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_structured_dataB
Read-only

Extract structured data using CSS selectors or LLM. Use output_path to persist the full extraction (including table_data) to disk as JSON and receive a slim response.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesTarget URL
extraction_typeNo'css'|'llm'|'table'css
css_selectorsNoCSS selector mapping
extraction_schemaNoSchema definition
generate_markdownNoGenerate markdown
wait_for_jsNoWait for JavaScript
timeoutNoTimeout in seconds
use_llm_table_extractionNoUse LLM table extraction
table_chunking_strategyNo'intelligent'|'fixed'|'semantic'intelligent
output_pathNoAbsolute file path (auto .json extension) to persist the full extracted_data + table_data as JSON. When set, the response is slimmed (content, markdown, table_data, extracted_data.raw_content removed).
include_content_in_responseNoWhen True (with output_path set), also keep extracted_data/table_data/content in the response. Defaults to False.
overwriteNoOverwrite an existing output file at output_path. Defaults to False.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations indicate readOnlyHint=true, which matches the extraction purpose. The description adds useful behavioral context about output_path persisting full data and slimming the response, but does not disclose other behaviors like rate limits or auth needs.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise with two sentences, no wasted words. It front-loads the main purpose, but could be better structured by separating the core function from the output_path detail.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (12 parameters) and presence of an output schema, the description covers the key behavior of output_path but lacks context on when to use different extraction types or how they differ. This is adequate but not comprehensive.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents parameters thoroughly. The description adds minimal value beyond restating extraction types and output_path behavior, meeting the baseline expectation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states the tool extracts structured data using CSS selectors or LLM, which is specific and clear. It distinguishes from sibling crawling tools but does not explicitly differentiate from other extraction tools like intelligent_extract or extract_entities.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus alternatives. It mentions the output_path behavior but lacks prerequisites or when-not-to-use instructions, which is a gap given the large set of sibling extraction tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_youtube_commentsB
Read-only

Extract YouTube video comments. Supports pagination via comment_offset. Use output_path to persist the full unsliced comment list to disk as JSON; the response is then slimmed to metadata only.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesYouTube video URL
sort_byNo'popular'|'recent'popular
max_commentsNoMax comments to retrieve (1-1000)
comment_offsetNoNumber of comments to skip (for pagination)
include_repliesNoInclude reply comments
content_offsetNoStart position for content (0-indexed)
content_limitNoMax characters to return (0=unlimited)
output_pathNoAbsolute file path (auto .json extension) to persist the full unsliced/untruncated comment list. When set, the complete comment array is written to disk BEFORE the internal token-limit reduction, and the response is slimmed (extracted_data.comments removed but comment_count etc. kept).
include_content_in_responseNoWhen True (with output_path set), keep the comment list in the response too. Note: the response copy is still subject to content_limit/content_offset slicing and the token-limit comment-array reduction; only the on-disk file holds the full list. Defaults to False.
overwriteNoOverwrite an existing output file at output_path. Defaults to False.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.3/5.0
Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Description describes a side effect (writing comments to disk) that contradicts the readOnlyHint annotation, which implies no mutations. This is a serious inconsistency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences; first states core function, second explains key persistence feature. No superfluous information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers main behavior, pagination, and file persistence, but the annotation contradiction undermines completeness. Output schema exists to cover return values.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Input schema has 100% description coverage, so baseline 3. Description adds marginal value beyond schema, clarifying output_path behavior but not essential.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states verb 'Extract' and resource 'YouTube video comments'. Differentiates from sibling tools like extract_youtube_transcript.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Implies usage for extracting comments with pagination and optional file persistence, but lacks explicit when-to-use or alternative comparisons.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

extract_youtube_transcriptA
Read-only

Extract YouTube transcripts with timestamps. Works with public captioned videos. Supports fallback to page crawl. Use output_path to persist the full unsliced transcript to disk as markdown.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesYouTube video URL
languagesNoLanguage codes in preference order
translate_toNoTarget language for translation
include_timestampsNoInclude timestamps
preserve_formattingNoPreserve formatting
include_metadataNoInclude video metadata
auto_summarizeNoAuto-summarize large content
max_content_tokensNoMax tokens before summarization
summary_lengthNo'short'|'medium'|'long'medium
llm_providerNoLLM provider
llm_modelNoLLM model
enable_crawl_fallbackNoEnable page crawl fallback when API fails
fallback_timeoutNoFallback crawl timeout in seconds
enrich_metadataNoEnrich metadata (upload_date, view_count) via page crawl
content_offsetNoStart position for content (0-indexed)
content_limitNoMax characters to return (0=unlimited)
output_pathNoAbsolute file path (auto .md extension) to persist the full unsliced transcript. When set, the response is slimmed to metadata+file path. content_limit/content_offset still affect the response copy but not the on-disk file.
include_content_in_responseNoWhen True (with output_path set), keep the transcript in the response too. Note: the response copy is still subject to content_limit/content_offset slicing; only the on-disk file holds the full transcript. Defaults to False.
overwriteNoOverwrite an existing output file at output_path. Defaults to False.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds value beyond annotations (readOnlyHint: true) by disclosing fallback crawling behavior, output_path persistence, metadata enrichment, and auto-summarization. No contradictions found. Could mention rate limits or video length constraints.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four sentences, front-loaded with the core purpose. Each sentence adds distinct information: extraction, constraints, fallback, output guidance. Slightly verbose but no redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (19 params, output schema exists), the description covers key behaviors: extraction, fallback, output persistence, summarization, metadata enrichment. It omits rate limits and language constraints but the schema covers languages. Output schema reduces need to describe return values.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

All 19 parameters have schema descriptions (100% coverage), and the description adds context for output_path (persist full unsliced transcript as markdown) and fallback. The description does not override schema but complements it effectively.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool extracts YouTube transcripts with timestamps, specifying the resource (YouTube) and action (extract). It distinguishes from siblings like get_youtube_video_info (video metadata) and batch_extract_youtube_transcripts (batch variant).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides context: works only with public captioned videos, supports fallback to page crawl, and suggests using output_path for persistence. However, it does not explicitly compare to sibling tools like extract_youtube_comments or state when to avoid using this tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_search_genresA
Read-only

Get available search genres for targeted searching.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and openWorldHint=false. The description adds no further behavioral details such as whether the genre list is static or dynamic, or any rate limits. It is minimally adequate given the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence with the verb 'Get' front-loaded, containing no unnecessary words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple tool with no parameters and an output schema, the description sufficiently conveys the tool's purpose. The agent can rely on the output schema for return structure details.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has zero parameters, so the description does not need to explain them. The baseline score of 4 is appropriate as the description does not contradict or fail to add value.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool retrieves available search genres for targeted searching, using a specific verb ('Get') and resource ('search genres'). It is distinct from sibling tools that perform searches themselves.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage when needing genre options for a targeted search, but does not explicitly state when not to use it or provide alternatives among siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_supported_file_formatsA
Read-only

Get supported file formats (PDF, Office, ZIP) and their capabilities.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations indicate readOnlyHint=true. The description adds context by specifying it returns format capabilities, which goes beyond the annotation. No contradictions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence that front-loads the action and examples. No extraneous words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given zero parameters and the presence of an output schema, the description sufficiently informs the agent about the tool's purpose and return value.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There are no parameters, so the description does not need to add meaning beyond the input schema. The schema is fully covered.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool retrieves supported file formats and their capabilities, with specific examples (PDF, Office, ZIP). It is distinct from sibling tools which focus on crawling, extraction, or processing.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The context is clear: this tool is used to check which formats are supported before using other file-processing tools. No explicit exclusions or alternatives are needed given the tool's simplicity.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_youtube_video_infoA
Read-only

Get YouTube video metadata and transcript availability. Use output_path to persist the full transcript to disk as markdown and receive a slim response.

ParametersJSON Schema
NameRequiredDescriptionDefault
video_urlYesYouTube video URL
summarize_transcriptNoSummarize transcript
max_tokensNoToken limit for summarization
llm_providerNoLLM provider
llm_modelNoLLM model
summary_lengthNo'short'|'medium'|'long'medium
include_timestampsNoInclude timestamps
output_pathNoAbsolute file path (auto .md extension) to persist the full video info + transcript as markdown. When set, the response is slimmed to metadata+file path.
include_content_in_responseNoWhen True (with output_path set), also include the full transcript in the response. Defaults to False.
overwriteNoOverwrite an existing output file at output_path. Defaults to False.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations declare readOnlyHint=true. The description adds that persisting to disk writes to a local file and alters response format. This provides behavioral context beyond the annotation, though the side effect of file writing is still safe.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences, front-loaded with purpose and key option. No unnecessary words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Description covers core purpose and one key parameter, but with 10 parameters and sibling tools, more context on summarization and differentiation would be beneficial. Output schema covers return values, so not necessary in description.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so parameters are well-documented. The description adds value for output_path by explaining its effect on response, but other parameters like summarize_transcript are not elaborated beyond schema defaults.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool retrieves video metadata and transcript availability, and mentions persisting transcript to disk. Verb 'Get' is specific. However, it does not explicitly differentiate from sibling tools like extract_youtube_transcript.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description hints at using output_path but provides no guidance on when to use this tool versus alternatives like extract_youtube_transcript. No context on exclusions or prerequisites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

intelligent_extractA
Read-only

Extract specific data from web pages using LLM. Use output_path to persist the full extraction output to disk as JSON and receive a slim response.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesTarget URL
extraction_goalYesData to extract
content_filterNo'bm25'|'pruning'|'llm'bm25
filter_queryNoBM25 filter keywords
chunk_contentNoSplit content
use_llmNoEnable LLM
llm_providerNoLLM provider
llm_modelNoLLM model
custom_instructionsNoLLM instructions
output_pathNoAbsolute file path (auto .json extension) to persist the full extracted data + content as JSON. When set, the response is slimmed to metadata+file path (extracted_data.raw_content, content, markdown, table_data removed).
include_content_in_responseNoWhen True (with output_path set), also keep extracted_data/content in the response. Defaults to False.
overwriteNoOverwrite an existing output file at output_path. Defaults to False.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations indicate readOnlyHint=true, so the tool is non-destructive. The description adds the behavior of output_path enabling slim responses. No further details on rate limits, error handling, or LLM behavior are provided. Does not contradict annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences: first states purpose, second explains key workflow parameter. No extraneous information. Efficient and front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (12 parameters, output schema present), the description covers the main purpose and the critical output_path behavior. It could be improved by clarifying when to use this over crawling or batch tools, but overall it is sufficient for an agent to understand the tool's role.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so most parameter meanings are already clear. The description adds context for output_path, explaining its effect on response structure. This provides marginal additional value beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool extracts specific data from web pages using LLM, distinguishing it from sibling tools like crawl_url which likely return full content. The verb 'extract' and resource 'web pages' are specific.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for extracting data from web pages with optional output persistence, but does not explicitly state when to use this tool versus alternatives like batch_crawl or search_and_crawl. No exclusions or prerequisites are mentioned.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

multi_url_crawlA
Read-only

Multi-URL crawl with pattern-based config. Max 5 URL patterns per call. Use output_path (directory) to persist full per-URL markdown + index.json; the return shape stays a list, each success item gets an output_file key.

ParametersJSON Schema
NameRequiredDescriptionDefault
url_configurationsYesURL-config mapping (max 5 URLs). Example: {'https://site1.com': {'wait_for_js': true}}
pattern_matchingNoPattern: 'wildcard' or 'regex' (default: wildcard)wildcard
default_configNoDefault config
base_timeoutNoTimeout per URL (default: 30)
max_concurrentNoMax concurrent (default: 3)
output_pathNoAbsolute directory path to persist per-URL markdown files + index.json. Existing regular files at this path are rejected; otherwise the directory is created if missing (dot-containing names are fine). The list return shape is preserved; each successful item gains an 'output_file' key. Failed items (success=False) are NOT written as .md but still appear in index.json with file=null.
include_content_in_responseNoWhen True (with output_path), keep full markdown/content in each list item. Defaults to False.
overwriteNoOverwrite existing per-URL files inside output_path. Defaults to False.

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description details output_path behavior: persisting per-URL markdown + index.json, return shape preservation, output_file key, and handling of failed items. This adds significant value beyond annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences with no wasted words, front-loaded with action and key constraints. Highly efficient.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given complexity (8 parameters, nested objects, output schema), the description covers core behavior, output path details, and return shape. Output schema exists, so return values are explained elsewhere.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, but the description adds constraints like max 5 URLs and clarifies return shape, which are not in schema. This provides meaningful extra context.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states it performs multi-URL crawls with pattern-based config, distinguishing it from single-URL tools like crawl_url. It specifies max 5 URL patterns and mentions output persistence, which is unique.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies use for multiple URLs with pattern matching and mentions max 5 patterns. It does not explicitly exclude other tools but provides enough context for selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

process_fileB
Read-only

Convert PDF, Word, Excel, PowerPoint, ZIP to markdown. Use output_path to persist the full unsliced converted markdown to disk and receive a slim response.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYesFile URL or local path (PDF, Office, ZIP). Supports http/https URLs, file:// URIs, and absolute paths.
max_size_mbNoMax file size in MB
extract_all_from_zipNoExtract ZIP contents
include_metadataNoInclude metadata
auto_summarizeNoAuto-summarize large content
max_content_tokensNoMax tokens before summarization
summary_lengthNo'short'|'medium'|'long'medium
llm_providerNoLLM provider
llm_modelNoLLM model
content_limitNoMax characters to return (0=unlimited)
content_offsetNoStart position for content (0-indexed)
output_pathNoAbsolute file path (auto .md extension) to persist the full unsliced converted markdown. When set, the response is slimmed to metadata+file path. content_limit/content_offset still affect the response copy but not the on-disk file.
include_content_in_responseNoWhen True (with output_path set), keep content in the response too. Note: the response copy is still subject to content_limit/content_offset slicing; only the on-disk file holds the full unsliced payload. Defaults to False.
overwriteNoOverwrite an existing output file at output_path. Defaults to False.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

B3.4/5.0
Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description states the tool 'converts' files, implying a write/mutation operation, but the annotation declares readOnlyHint=true. This is a direct contradiction and severely misleads the agent about the tool's side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences: first states core purpose, second explains key usage pattern. Every word is necessary and front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Essentials covered: conversion types and output_path. With full schema coverage and output schema present, the description is sufficiently complete for a complex tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, baseline 3. The description adds value by explaining the output_path behavior (slim response, full unsliced disk persistence) and slicing effect, going beyond schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool converts PDF, Word, Excel, PowerPoint, ZIP to markdown, and mentions the key output_path feature. It is specific and informative.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance on when to use this tool versus siblings like batch_crawl or crawl_url. The description does not mention alternatives or exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_and_crawlA
Read-only

Search Google and crawl top results. Combines search with full content extraction. Supply output_path (directory) in the request to persist per-page markdown (unsliced) + index.json and receive a slim response.

ParametersJSON Schema
NameRequiredDescriptionDefault
requestYesDict with: search_query (required), crawl_top_results, search_genre, recent_days, generate_markdown, max_content_per_page. Optional persistence keys: output_path (absolute directory — per-page .md files + index.json, the full page bodies are written BEFORE max_content_per_page truncation; dot-containing dir names are fine), include_content_in_response (bool, default False — when True keeps crawled_pages in the response too, still subject to max_content_per_page truncation), overwrite (bool, default False). Failed pages appear in index.json with file=null.

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A3.7/5.0
Behavior1/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations set readOnlyHint=true, but the description describes writing files to disk (per-page markdown, index.json), which is a side effect. This is a direct contradiction. The description does not acknowledge or clarify the discrepancy, nor does it explain any non-destructive nature.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is succinct with only two sentences, yet covers the core purpose, output mechanism, and key configurations. Every sentence adds essential information; no redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite the complexity of combining search and crawl with many options, the description covers input, persistence, response behavior, and error handling. However, it omits details on rate limiting, timeouts, or the output schema structure, which would enhance completeness.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The parameter description inside the input schema is highly detailed, explaining each key's purpose, default behavior (e.g., 'full page bodies are written BEFORE truncation'), and special cases (failed pages). This adds significant value beyond the schema structure, achieving high clarity.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool combines Google search with crawling top results, specifying verbs 'Search' and 'crawl' on the resource 'Google results'. It distinctively differentiates from siblings like 'search_google' and 'crawl_url' by highlighting the combination.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides instructions on when to use (to combine search and crawl with persistence) but lacks explicit guidance on when not to use it or alternatives. No mention of siblings or trade-offs.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

search_googleA
Read-only

Search Google with genre filtering. Genres: academic, news, technical, commercial, social. Supply output_path in the request to persist the full unsliced result set to disk as JSON and receive a slim response.

ParametersJSON Schema
NameRequiredDescriptionDefault
requestYesDict with: query (required), num_results, search_genre, language, region, recent_days, content_limit (int), content_offset (int). Optional persistence keys: output_path (absolute file path, auto .json extension — full unsliced results written to disk BEFORE content_limit/content_offset slicing), include_content_in_response (bool, default False — when True keeps results in the response too, still subject to slicing), overwrite (bool, default False).

Output Schema

ParametersJSON Schema
NameRequiredDescription

No output parameters

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations declare readOnlyHint:true. Description adds optional disk persistence via output_path and slim response behavior, providing useful context beyond annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with purpose and genres, and a second sentence for persistence. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the existence of an output schema, the description adequately covers genre filtering and optional persistence. It omits details on result slicing but the schema covers that.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with detailed descriptions for all sub-parameters. The tool description reinforces the output_path key but adds no new meaning beyond existing schema documentation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Search Google with genre filtering', listing specific genres. This distinguishes it from siblings like batch_search_google or search_and_crawl.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies use for single Google searches with genre filtering but does not explicitly compare to sibling tools or state when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

TDQS

A3.9/5.0
Disambiguation4/5

Most tools have distinct purposes, but there is potential overlap between batch_crawl and multi_url_crawl, and between single/batch versions of search and YouTube extraction. Descriptions help disambiguate, but slight confusion may occur.

Naming Consistency5/5

All tools follow a consistent verb_noun snake_case pattern, with batch_ prefix for batch operations and get_ prefix for metadata retrieval. No mixing of conventions.

Tool Count5/5

19 tools cover a broad domain (web crawling, search, YouTube, file processing, extraction) without being excessive. Each tool has a clear role and the count feels well-scoped.

Completeness5/5

The tool set covers the full lifecycle of web data extraction: crawling (single, batch, deep), search, YouTube operations, file conversion, and multiple extraction methods (entities, structured, intelligent). No obvious gaps for the stated purpose.

Maintenance

ActivitySlowing
ResponsivenessSyncing

Resources

Unclaimed servers have limited discoverability.

Looking for Admin?

If you are the server author, to access and configure the admin panel.

Related MCP Connectors

Related MCP Servers

  • F
    license
    Not graded
    quality
    D
    maintenance
    A locally-hosted MCP server that provides AI assistants with advanced web crawling capabilities, including structured data extraction, deep site crawling, and page screenshots. It enables users to convert single or multiple URLs into clean Markdown content for processing by LLMs without requiring external API keys for basic features.
  • A
    license
    Not graded
    quality
    C
    maintenance
    MCP server integrating Crawl4AI for universal web crawling and data extraction. Enables AI agents to crawl, extract markdown/HTML, take screenshots, generate PDFs, and execute JavaScript on web pages.
    33
    6
    MIT

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/walksoda/crawl-mcp'

If you have feedback or need assistance with the MCP directory API, please join our Discord server