Crawl-MCP
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| CRAWL4AI_LANG | No | The language for the interface (e.g., 'en' for English, 'ja' for Japanese) | en |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
This server publishes no instructions, or was last inspected before Glama recorded them.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tasks | {
"list": {},
"cancel": {},
"requests": {
"tools": {
"call": {}
},
"prompts": {
"get": {}
},
"resources": {
"read": {}
}
}
} |
| tools | {
"listChanged": true
} |
| prompts | {
"listChanged": false
} |
| resources | {
"subscribe": false,
"listChanged": false
} |
| experimental | {} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| crawl_urlA | Extract web page content with JavaScript support. Use wait_for_js=true for SPAs. Use content_offset/content_limit to paginate the response. Use output_path to persist the full unsliced content to disk as markdown and receive a slim metadata-only response. |
| deep_crawl_siteA | Crawl multiple pages from a site with configurable depth. Use output_path (directory) to persist per-URL markdown files + index.json; the response is then slimmed to metadata only. |
| crawl_url_with_fallbackA | Crawl with fallback strategies for anti-bot sites. Use content_offset/content_limit to paginate the response. Use output_path to persist the full unsliced content to disk as markdown and receive a slim response. |
| intelligent_extractA | Extract specific data from web pages using LLM. Use output_path to persist the full extraction output to disk as JSON and receive a slim response. |
| extract_entitiesB | Extract entities (emails, phones, etc.) from web pages. Use output_path to persist the full entity extraction output to disk as JSON and receive a slim response. |
| extract_structured_dataB | Extract structured data using CSS selectors or LLM. Use output_path to persist the full extraction (including table_data) to disk as JSON and receive a slim response. |
| extract_youtube_transcriptA | Extract YouTube transcripts with timestamps. Works with public captioned videos. Supports fallback to page crawl. Use output_path to persist the full unsliced transcript to disk as markdown. |
| batch_extract_youtube_transcriptsA | Extract transcripts from multiple YouTube videos. Max 3 URLs per call. Supply output_path (directory) in the request to persist per-video markdown files + index.json and receive a slim response. |
| get_youtube_video_infoA | Get YouTube video metadata and transcript availability. Use output_path to persist the full transcript to disk as markdown and receive a slim response. |
| extract_youtube_commentsB | Extract YouTube video comments. Supports pagination via comment_offset. Use output_path to persist the full unsliced comment list to disk as JSON; the response is then slimmed to metadata only. |
| process_fileB | Convert PDF, Word, Excel, PowerPoint, ZIP to markdown. Use output_path to persist the full unsliced converted markdown to disk and receive a slim response. |
| get_supported_file_formatsA | Get supported file formats (PDF, Office, ZIP) and their capabilities. |
| enhanced_process_large_contentB | Process large content with chunking and BM25 filtering. Use output_path to persist chunks + summaries to disk as JSON and receive a slim response. |
| search_webA | Search the web with Google Custom Search, Brave, Exa or SearXNG (engine key, or the server default). Optional search_genre filtering (file types or language; see get_search_genres). Supply output_path in the request to persist the full unsliced result set to disk as JSON and receive a slim response. |
| search_googleA | Deprecated alias of search_web (same request keys and results; the engine is not limited to Google). Use search_web. |
| batch_search_webA | Perform multiple web searches. Max 3 queries per call. Supply output_path in the request to persist the full result set to disk as JSON and receive a slim response. |
| batch_search_googleA | Deprecated alias of batch_search_web (same request keys and results). Use batch_search_web. |
| search_and_crawlA | Search the web (engine as in search_web) and crawl top results. Combines search with full content extraction. Supply output_path (directory) in the request to persist per-page markdown (unsliced) + index.json and receive a slim response. |
| get_search_genresA | Get available search genres for targeted searching. |
| batch_crawlA | Crawl multiple URLs with fallback. Max 3 URLs per call. Use output_path (directory) to persist full per-URL markdown + index.json; the return shape stays a list, each success item gets an output_file key. |
| multi_url_crawlB | Multi-URL crawl with pattern-based config. Max 5 URL patterns per call. Use output_path (directory) to persist full per-URL markdown + index.json; the return shape stays a list, each success item gets an output_file key. |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 21 tools
Several tools have overlapping boundaries: crawl_url vs crawl_url_with_fallback (same operation with fallback strategies), batch_crawl vs multi_url_crawl (both crawl multiple URLs, differing only in limits/config style), and extract_structured_data vs intelligent_extract (both LLM-based extraction). The deprecated search_google/batch_search_google aliases add further surface duplication, though descriptions do flag them as aliases and help disambiguate.
All names use snake_case and mostly follow a verb_noun pattern (crawl_url, extract_entities, search_web, get_youtube_video_info). A few deviate to adjective-first phrasing (enhanced_process_large_content, intelligent_extract), which is a minor inconsistency but still readable and predictable.
21 tools is on the heavy side for a crawling/search server, and the count is inflated by two deprecated aliases plus redundant crawl variants (batch_crawl/multi_url_crawl, crawl_url/crawl_url_with_fallback). The core capability set is reasonable, but pruning the aliases and merged variants would tighten the surface.
Coverage is broad and coherent: search (single/batch/genres), crawling (single, batch, multi-URL, deep site, fallback), extraction (entities, structured data, LLM), YouTube (info, transcript, comments, batch), and file conversion, plus format/genre discovery tools. Minor gaps like sitemap/robots handling or cache/session management exist but agents can work around them.