Crawl4AI MCP Server

A comprehensive Model Context Protocol (MCP) server that wraps the powerful crawl4ai library. This server provides advanced web crawling, content extraction, and AI-powered analysis capabilities through the standardized MCP interface.

🚀 Key Features

Core Capabilities

🚀 Complete JavaScript Support

This feature set enables comprehensive JavaScript-heavy website handling:

• Full Playwright Integration - React, Vue, Angular SPA sites fully supported

• Dynamic Content Loading - Auto-waits for content to load

• Custom JavaScript Execution - Run custom scripts on pages

• DOM Element Waiting - wait_for_selector for specific elements

• Human-like Browsing Simulation - Bypass basic anti-bot measures

JavaScript-Heavy Sites Recommended Settings:

{
  "wait_for_js": true,
  "simulate_user": true, 
  "timeout": 30-60,
  "generate_markdown": true
}

• Advanced Web Crawling with complete JavaScript execution support

• Deep Crawling with configurable depth and multiple strategies (BFS, DFS, Best-First)

• AI-Powered Content Extraction using LLM-based analysis

• 📄 File Processing with Microsoft MarkItDown integration

  • PDF, Office documents, ZIP archives, and more

  • Automatic file format detection and conversion

  • Batch processing of archive contents

• 📺 YouTube Transcript Extraction (youtube-transcript-api v1.1.0+)

  • No authentication required - works out of the box

  • Stable and reliable transcript extraction

  • Support for both auto-generated and manual captions

  • Multi-language support with priority settings

  • Timestamped segment information and clean text output

  • Batch processing for multiple videos

• Entity Extraction with 9 built-in patterns including emails, phones, URLs, and dates

• Intelligent Content Filtering (BM25, pruning, LLM-based)

• Content Chunking for large document processing

• Screenshot Capture and media extraction

Advanced Features

• 🔍 Google Search Integration with genre-based filtering and metadata extraction

  • 31 search genres (academic, programming, news, etc.)

  • Automatic title and snippet extraction from search results

  • Safe search enabled by default for security

  • Batch search capabilities with result analysis

• Multiple Extraction Strategies include CSS selectors, XPath, regex patterns, and LLM-based extraction

• Browser Automation supports custom user agents, headers, cookies, and authentication

• Caching System with multiple modes for performance optimization

• Custom JavaScript Execution for dynamic content interaction

• Structured Data Export in multiple formats (JSON, Markdown, HTML)

📦 Installation

Quick Setup

Linux/macOS:

./setup.sh

Windows:

setup_windows.bat

Manual Installation

1. Create and activate virtual environment:

python3 -m venv venv
source venv/bin/activate  # Linux/macOS
# or
venv\Scripts\activate.bat  # Windows

2. Install Python dependencies:

pip install -r requirements.txt

3. Install Playwright browser dependencies (Linux/WSL):

sudo apt-get update
sudo apt-get install libnss3 libnspr4 libasound2 libatk-bridge2.0-0 libdrm2 libgtk-3-0 libgbm1

🐳 Docker Deployment (Recommended)

For production deployments and easy setup, we provide multiple Docker container variants optimized for different use cases:

Quick Start with Docker

# Create shared network
docker network create shared_net

# CPU-optimized deployment (recommended for most users)
docker-compose -f docker-compose.cpu.yml build
docker-compose -f docker-compose.cpu.yml up -d

# Verify deployment
docker-compose -f docker-compose.cpu.yml ps

Container Variants

Docker Features

• 🔒 Network Isolation: No localhost exposure, only accessible via shared_net

• ⚡ CPU Optimization: CUDA-free builds for 50-70% smaller containers

• 🔄 Auto-scaling: RunPod serverless deployment with automated CI/CD

• 📊 Health Monitoring: Built-in health checks and monitoring

• 🛡️ Security: Non-root user, resource limits, safe mode enabled

Quick Commands

# Build all variants for comparison
./scripts/build-cpu-containers.sh

# Deploy to RunPod (automated via GitHub Actions)
# Image: docker.io/gemneye/crawl4ai-runpod-serverless:latest

# Health check
docker exec crawl4ai-mcp-cpu python -c "from crawl4ai_mcp.server import mcp; print('✅ Healthy')"

For complete Docker documentation, see Docker Guide.

🖥️ Usage

Start the MCP Server

STDIO transport (default):

python -m crawl4ai_mcp.server

HTTP transport:

python -m crawl4ai_mcp.server --transport http --host 127.0.0.1 --port 8000

📋 MCP Command Registration (Claude Code CLI)

You can register this MCP server with Claude Code CLI. The following methods are available:

Using .mcp.json Configuration (Recommended)

1. Create or update .mcp.json in your project directory:

{
  "mcpServers": {
    "crawl4ai": {
      "command": "/home/user/prj/crawl/venv/bin/python",
      "args": ["-m", "crawl4ai_mcp.server"],
      "env": {
        "FASTMCP_LOG_LEVEL": "DEBUG"
      }
    }
  }
}

2. Run claude mcp or start Claude Code from the project directory

Alternative: Command Line Registration

# Register the MCP server with claude command
claude mcp add crawl4ai "/path/to/your/venv/bin/python -m crawl4ai_mcp.server" \
  --cwd /path/to/your/crawl4ai-mcp-project

# With environment variables
claude mcp add crawl4ai "/path/to/your/venv/bin/python -m crawl4ai_mcp.server" \
  --cwd /path/to/your/crawl4ai-mcp-project \
  -e FASTMCP_LOG_LEVEL=DEBUG

# With project scope (shared with team)
claude mcp add crawl4ai "/path/to/your/venv/bin/python -m crawl4ai_mcp.server" \
  --cwd /path/to/your/crawl4ai-mcp-project \
  --scope project

HTTP Transport (For Remote Access)

# First start the HTTP server
python -m crawl4ai_mcp.server --transport http --host 127.0.0.1 --port 8000

# Then register the HTTP endpoint
claude mcp add crawl4ai-http --transport http --url http://127.0.0.1:8000/mcp

# Or with Pure StreamableHTTP (recommended)
./scripts/start_pure_http_server.sh
claude mcp add crawl4ai-pure-http --transport http --url http://127.0.0.1:8000/mcp

Verification

# List registered MCP servers
claude mcp list

# Test the connection
claude mcp test crawl4ai

# Remove if needed
claude mcp remove crawl4ai

Setting API Keys (Optional for LLM Features)

# Add with environment variables for LLM functionality
claude mcp add crawl4ai "python -m crawl4ai_mcp.server" \
  --cwd /path/to/your/crawl4ai-mcp-project \
  -e OPENAI_API_KEY=your_openai_key \
  -e ANTHROPIC_API_KEY=your_anthropic_key

Claude Desktop Integration

🎯 Pure StreamableHTTP Usage (Recommended)

1. Start Server by running the startup script:

   ./scripts/start_pure_http_server.sh

2. Apply Configuration using one of these methods:

   • Copy configs/claude_desktop_config_pure_http.json to Claude Desktop's config directory

   • Or add the following to your existing config:

   {
     "mcpServers": {
       "crawl4ai-pure-http": {
         "url": "http://127.0.0.1:8000/mcp"
       }
     }
   }

3. Restart Claude Desktop to apply settings

4. Start Using the tools - crawl4ai tools are now available in chat

🔄 Traditional STDIO Usage

1. Copy the configuration:

   cp configs/claude_desktop_config.json ~/.config/claude-desktop/claude_desktop_config.json

2. Restart Claude Desktop to enable the crawl4ai tools

📂 Configuration File Locations

Windows:

%APPDATA%\Claude\claude_desktop_config.json

macOS:

~/Library/Application Support/Claude/claude_desktop_config.json

Linux:

~/.config/claude-desktop/claude_desktop_config.json

🌐 HTTP API Access

This MCP server supports multiple HTTP protocols, allowing you to choose the optimal implementation for your use case.

🎯 Pure StreamableHTTP (Recommended)

Pure JSON HTTP protocol without Server-Sent Events (SSE)

Server Startup

# Method 1: Using startup script
./scripts/start_pure_http_server.sh

# Method 2: Direct startup
python examples/simple_pure_http_server.py --host 127.0.0.1 --port 8000

# Method 3: Background startup
nohup python examples/simple_pure_http_server.py --port 8000 > server.log 2>&1 &

Claude Desktop Configuration

{
  "mcpServers": {
    "crawl4ai-pure-http": {
      "url": "http://127.0.0.1:8000/mcp"
    }
  }
}

Usage Steps

1. Start Server: ./scripts/start_pure_http_server.sh

2. Apply Configuration: Use configs/claude_desktop_config_pure_http.json

3. Restart Claude Desktop: Apply settings

Verification

# Health check
curl http://127.0.0.1:8000/health

# Complete test
python examples/pure_http_test.py

🔄 Legacy HTTP (SSE Implementation)

Traditional FastMCP StreamableHTTP protocol (with SSE)

Server Startup

# Method 1: Command line
python -m crawl4ai_mcp.server --transport http --host 127.0.0.1 --port 8001

# Method 2: Environment variables
export MCP_TRANSPORT=http
export MCP_HOST=127.0.0.1
export MCP_PORT=8001
python -m crawl4ai_mcp.server

Claude Desktop Configuration

{
  "mcpServers": {
    "crawl4ai-legacy-http": {
      "url": "http://127.0.0.1:8001/mcp"
    }
  }
}

📊 Protocol Comparison

🚀 HTTP Usage Examples

Pure StreamableHTTP

# Initialize
SESSION_ID=$(curl -s -X POST http://127.0.0.1:8000/mcp/initialize \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":"init","method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1.0.0"}}}' \
  -D- | grep -i mcp-session-id | cut -d' ' -f2 | tr -d '\r')

# Execute tool
curl -X POST http://127.0.0.1:8000/mcp \
  -H "Content-Type: application/json" \
  -H "mcp-session-id: $SESSION_ID" \
  -d '{"jsonrpc":"2.0","id":"crawl","method":"tools/call","params":{"name":"crawl_url","arguments":{"url":"https://example.com"}}}'

Legacy HTTP

curl -X POST "http://127.0.0.1:8001/tools/crawl_url" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "generate_markdown": true}'

📚 Detailed Documentation

• Pure StreamableHTTP: docs/PURE_STREAMABLE_HTTP.md

• HTTP Server Usage: docs/HTTP_SERVER_USAGE.md

• Legacy HTTP API: docs/HTTP_API_GUIDE.md

Starting the HTTP Server

Method 1: Command Line

python -m crawl4ai_mcp.server --transport http --host 127.0.0.1 --port 8000

Method 2: Environment Variables

export MCP_TRANSPORT=http
export MCP_HOST=127.0.0.1
export MCP_PORT=8000
python -m crawl4ai_mcp.server

Method 3: Docker (if available)

docker run -p 8000:8000 crawl4ai-mcp --transport http --port 8000

Basic Endpoint Information

Once running, the HTTP API provides:

• Base URL: http://127.0.0.1:8000

• OpenAPI Documentation: http://127.0.0.1:8000/docs

• Tool Endpoints: http://127.0.0.1:8000/tools/{tool_name}

• Resource Endpoints: http://127.0.0.1:8000/resources/{resource_uri}

All MCP tools (crawl_url, intelligent_extract, process_file, etc.) are accessible via HTTP POST requests with JSON payloads matching the tool parameters.

Quick HTTP Example

curl -X POST "http://127.0.0.1:8000/tools/crawl_url" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "generate_markdown": true}'

For detailed HTTP API documentation, examples, and integration guides, see the HTTP API Guide.

🛠️ Tool Selection Guide

📋 Choose the Right Tool for Your Task

⚡ Performance Guidelines

• Deep Crawling: Limited to 5 pages max (stability focused)

• Batch Processing: Concurrent limits enforced

• Timeout Calculation: pages × base_timeout recommended

• Large Files: 100MB maximum size limit

• Retry Strategy: Manual retry recommended on first failure

🎯 Best Practices

For JavaScript-Heavy Sites:

• Always use wait_for_js: true

• Set simulate_user: true for better compatibility

• Increase timeout to 30-60 seconds

• Use wait_for_selector for specific elements

For AI Features:

• Configure LLM settings with get_llm_config_info

• Fallback to non-AI tools if LLM unavailable

• Use intelligent_extract for semantic understanding

🛠️ MCP Tools

crawl_url

Advanced web crawling with deep crawling support and intelligent filtering.

Key Parameters:

• url: Target URL to crawl

• max_depth: Maximum crawling depth (None for single page)

• crawl_strategy: Strategy type ('bfs', 'dfs', 'best_first')

• content_filter: Filter type ('bm25', 'pruning', 'llm')

• chunk_content: Enable content chunking for large documents

• execute_js: Custom JavaScript code execution

• user_agent: Custom user agent string

• headers: Custom HTTP headers

• cookies: Authentication cookies

deep_crawl_site

Dedicated tool for comprehensive site mapping and recursive crawling.

Parameters:

• url: Starting URL

• max_depth: Maximum crawling depth (recommended: 1-3)

• max_pages: Maximum number of pages to crawl

• crawl_strategy: Crawling strategy ('bfs', 'dfs', 'best_first')

• url_pattern: URL filter pattern (e.g., 'docs', 'blog')

• score_threshold: Minimum relevance score (0.0-1.0)

intelligent_extract

AI-powered content extraction with advanced filtering and analysis.

Parameters:

• url: Target URL

• extraction_goal: Description of extraction target

• content_filter: Filter type for content quality

• use_llm: Enable LLM-based intelligent extraction

• llm_provider: LLM provider (openai, claude, etc.)

• custom_instructions: Detailed extraction instructions

extract_entities

High-speed entity extraction using regex patterns.

Built-in Entity Types:

• emails: Email addresses

• phones: Phone numbers

• urls: URLs and links

• dates: Date formats

• ips: IP addresses

• social_media: Social media handles (@username, #hashtag)

• prices: Price information

• credit_cards: Credit card numbers

• coordinates: Geographic coordinates

extract_structured_data

Traditional structured data extraction using CSS/XPath selectors or LLM schemas.

batch_crawl

Parallel processing of multiple URLs with unified reporting.

crawl_url_with_fallback

Robust crawling with multiple fallback strategies for maximum reliability.

process_file

📄 File Processing: Convert various file formats to Markdown using Microsoft MarkItDown.

Parameters:

• url: File URL (PDF, Office, ZIP, etc.)

• max_size_mb: Maximum file size limit (default: 100MB)

• extract_all_from_zip: Extract all files from ZIP archives

• include_metadata: Include file metadata in response

Supported Formats:

• PDF: .pdf

• Microsoft Office: .docx, .pptx, .xlsx, .xls

• Archives: .zip

• Web/Text: .html, .htm, .txt, .md, .csv, .rtf

• eBooks: .epub

get_supported_file_formats

📋 Format Information: Get comprehensive list of supported file formats and their capabilities.

extract_youtube_transcript

📺 YouTube Processing: Extract transcripts from YouTube videos with language preferences and translation using youtube-transcript-api v1.1.0+.

✅ Stable and reliable - No authentication required!

Parameters:

• url: YouTube video URL

• languages: Preferred languages in order of preference (default: ["ja", "en"])

• translate_to: Target language for translation (optional)

• include_timestamps: Include timestamps in transcript

• preserve_formatting: Preserve original formatting

• include_metadata: Include video metadata

batch_extract_youtube_transcripts

📺 Batch YouTube Processing: Extract transcripts from multiple YouTube videos in parallel.

✅ Enhanced performance with controlled concurrency for stable batch processing.

Parameters:

• urls: List of YouTube video URLs

• languages: Preferred languages list

• translate_to: Target language for translation (optional)

• include_timestamps: Include timestamps in transcript

• max_concurrent: Maximum concurrent requests (1-5, default: 3)

get_youtube_video_info

📋 YouTube Info: Get available transcript information for a YouTube video without extracting the full transcript.

Parameters:

• video_url: YouTube video URL

Returns:

• Available transcript languages

• Manual/auto-generated distinction

• Translatable language information

search_google

🔍 Google Search: Perform Google search with genre filtering and metadata extraction.

Parameters:

• query: Search query string

• num_results: Number of results to return (1-100, default: 10)

• language: Search language (default: "en")

• region: Search region (default: "us")

• search_genre: Content genre filter (optional)

• safe_search: Safe search enabled (always True for security)

Features:

• Automatic title and snippet extraction from search results

• 31 available search genres for content filtering

• URL classification and domain analysis

• Safe search enforced by default

batch_search_google

🔍 Batch Google Search: Perform multiple Google searches with comprehensive analysis.

Parameters:

• queries: List of search queries

• num_results_per_query: Results per query (1-100, default: 10)

• max_concurrent: Maximum concurrent searches (1-5, default: 3)

• language: Search language (default: "en")

• region: Search region (default: "us")

• search_genre: Content genre filter (optional)

Returns:

• Individual search results for each query

• Cross-query analysis and statistics

• Domain distribution and result type analysis

search_and_crawl

🔍 Integrated Search+Crawl: Perform Google search and automatically crawl top results.

Parameters:

• search_query: Google search query

• num_search_results: Number of search results (1-20, default: 5)

• crawl_top_results: Number of top results to crawl (1-10, default: 3)

• extract_media: Extract media from crawled pages

• generate_markdown: Generate markdown content

• search_genre: Content genre filter (optional)

Returns:

• Complete search metadata and crawled content

• Success rates and processing statistics

• Integrated analysis of search and crawl results

get_search_genres

📋 Search Genres: Get comprehensive list of available search genres and their descriptions.

Returns:

• 31 available search genres with descriptions

• Categorized genre lists (Academic, Technical, News, etc.)

• Usage examples for each genre type

📚 Resources

• uri://crawl4ai/config: Default crawler configuration options

• uri://crawl4ai/examples: Usage examples and sample requests

🎯 Prompts

• crawl_website_prompt: Guided website crawling workflows

• analyze_crawl_results_prompt: Crawl result analysis

• batch_crawl_setup_prompt: Batch crawling setup

🔧 Configuration Examples

🔍 Google Search Examples

Basic Google Search

{
    "query": "python machine learning tutorial",
    "num_results": 10,
    "language": "en",
    "region": "us"
}

Genre-Filtered Search

{
    "query": "machine learning research",
    "num_results": 15,
    "search_genre": "academic",
    "language": "en"
}

Batch Search with Analysis

{
    "queries": [
        "python programming tutorial",
        "web development guide", 
        "data science introduction"
    ],
    "num_results_per_query": 5,
    "max_concurrent": 3,
    "search_genre": "education"
}

Integrated Search and Crawl

{
    "search_query": "python official documentation",
    "num_search_results": 10,
    "crawl_top_results": 5,
    "extract_media": false,
    "generate_markdown": true,
    "search_genre": "documentation"
}

Basic Deep Crawling

{
    "url": "https://docs.example.com",
    "max_depth": 2,
    "max_pages": 20,
    "crawl_strategy": "bfs"
}

AI-Driven Content Extraction

{
    "url": "https://news.example.com",
    "extraction_goal": "article summary and key points",
    "content_filter": "llm",
    "use_llm": true,
    "custom_instructions": "Extract main article content, summarize key points, and identify important quotes"
}

📄 File Processing Examples

PDF Document Processing

{
    "url": "https://example.com/document.pdf",
    "max_size_mb": 50,
    "include_metadata": true
}

Office Document Processing

{
    "url": "https://example.com/report.docx",
    "max_size_mb": 25,
    "include_metadata": true
}

ZIP Archive Processing

{
    "url": "https://example.com/documents.zip",
    "max_size_mb": 100,
    "extract_all_from_zip": true,
    "include_metadata": true
}

Automatic File Detection

The crawl_url tool automatically detects file formats and routes to appropriate processing:

{
    "url": "https://example.com/mixed-content.pdf",
    "generate_markdown": true
}

📺 YouTube Video Processing Examples

✅ Stable youtube-transcript-api v1.1.0+ integration - No setup required!

Basic Transcript Extraction

{
    "url": "https://www.youtube.com/watch?v=VIDEO_ID",
    "languages": ["ja", "en"],
    "include_timestamps": true,
    "include_metadata": true
}

Auto-Translation Feature

{
    "url": "https://www.youtube.com/watch?v=VIDEO_ID",
    "languages": ["en"],
    "translate_to": "ja",
    "include_timestamps": false
}

Batch Video Processing

{
    "urls": [
        "https://www.youtube.com/watch?v=VIDEO_ID1",
        "https://www.youtube.com/watch?v=VIDEO_ID2",
        "https://youtu.be/VIDEO_ID3"
    ],
    "languages": ["ja", "en"],
    "max_concurrent": 3
}

Automatic YouTube Detection

The crawl_url tool automatically detects YouTube URLs and extracts transcripts:

{
    "url": "https://www.youtube.com/watch?v=VIDEO_ID",
    "generate_markdown": true
}

Video Information Lookup

{
    "video_url": "https://www.youtube.com/watch?v=VIDEO_ID"
}

Entity Extraction

{
    "url": "https://company.com/contact",
    "entity_types": ["emails", "phones", "social_media"],
    "include_context": true,
    "deduplicate": true
}

Authenticated Crawling

{
    "url": "https://private.example.com",
    "auth_token": "Bearer your-token",
    "cookies": {"session_id": "abc123"},
    "headers": {"X-API-Key": "your-key"}
}

🏗️ Project Structure

crawl4ai_mcp/
├── __init__.py              # Package initialization
├── server.py                # Main MCP server (1,184+ lines)
├── strategies.py            # Additional extraction strategies
└── suppress_output.py       # Output suppression utilities

config/
├── claude_desktop_config_windows.json  # Claude Desktop config (Windows)
├── claude_desktop_config_script.json   # Script-based config
└── claude_desktop_config.json          # Basic config

docs/
├── README_ja.md             # Japanese documentation
├── setup_instructions_ja.md # Detailed setup guide
└── troubleshooting_ja.md    # Troubleshooting guide

scripts/
├── setup.sh                 # Linux/macOS setup
├── setup_windows.bat        # Windows setup
└── run_server.sh            # Server startup script

🔍 Troubleshooting

Common Issues

ModuleNotFoundError:

• Ensure virtual environment is activated

• Verify PYTHONPATH is set correctly

• Install dependencies: pip install -r requirements.txt

Playwright Browser Errors:

• Install system dependencies: sudo apt-get install libnss3 libnspr4 libasound2

• For WSL: Ensure X11 forwarding or headless mode

JSON Parsing Errors:

• Resolved: Output suppression implemented in latest version

• All crawl4ai verbose output is now properly suppressed

For detailed troubleshooting, see docs/troubleshooting_ja.md.

📊 Supported Formats & Capabilities

✅ Web Content

• Static Sites: HTML, CSS, JavaScript

• Dynamic Sites: React, Vue, Angular SPAs

• Complex Sites: JavaScript-heavy, async loading

• Protected Sites: Basic auth, cookies, custom headers

✅ Media & Files

• Videos: YouTube (transcript auto-extraction)

• Documents: PDF, Word, Excel, PowerPoint, ZIP

• Archives: Automatic extraction and processing

• Text: Markdown, CSV, RTF, plain text

✅ Search & Data

• Google Search: 31 genre filters available

• Entity Extraction: Emails, phones, URLs, dates

• Structured Data: CSS/XPath/LLM-based extraction

• Batch Processing: Multiple URLs simultaneously

⚠️ Limitations & Important Notes

🚫 Known Limitations

• Authentication Sites: Cannot bypass login requirements

• reCAPTCHA Protected: Limited success on heavily protected sites

• Rate Limiting: Manual interval management recommended

• Automatic Retry: Not implemented - manual retry needed

• Deep Crawling: 5 page maximum for stability

🌐 Regional & Language Support

• Multi-language Sites: Full Unicode support

• Regional Search: Configurable region settings

• Character Encoding: Automatic detection

• Japanese Content: Complete support

🔄 Error Handling Strategy

1. First Failure → Immediate manual retry

2. Timeout Issues → Increase timeout settings

3. Persistent Problems → Use crawl_url_with_fallback

4. Alternative Approach → Try different tool selection

💡 Common Workflows

🔍 Research & Analysis

1. Competitive Analysis: search_and_crawl → intelligent_extract
2. Site Auditing: crawl_url → extract_entities  
3. Content Research: search_google → batch_crawl
4. Deep Analysis: deep_crawl_site → structured extraction

📈 Typical Success Patterns

• E-commerce Sites: Use simulate_user: true

• News Sites: Enable wait_for_js for dynamic content

• Documentation: Use deep_crawl_site with URL patterns

• Social Media: Extract entities for contact information

🚀 Performance Features

• Intelligent Caching: 15-minute self-cleaning cache with multiple modes

• Async Architecture: Built on asyncio for high performance

• Memory Management: Adaptive concurrency based on system resources

• Rate Limiting: Configurable delays and request throttling

• Parallel Processing: Concurrent crawling of multiple URLs

🛡️ Security Features

• Output Suppression provides complete isolation of crawl4ai output from MCP JSON

• Authentication Support includes token-based and cookie authentication

• Secure Headers offer custom header support for API access

• Error Isolation includes comprehensive error handling with helpful suggestions

📋 Dependencies

• crawl4ai>=0.3.0 - Advanced web crawling library

• fastmcp>=0.1.0 - MCP server framework

• pydantic>=2.0.0 - Data validation and serialization

• markitdown>=0.0.1a2 - File processing and conversion (Microsoft)

• googlesearch-python>=1.3.0 - Google search functionality

• aiohttp>=3.8.0 - Asynchronous HTTP client for metadata extraction

• beautifulsoup4>=4.12.0 - HTML parsing for title/snippet extraction

• youtube-transcript-api>=1.1.0 - Stable YouTube transcript extraction

• asyncio - Asynchronous programming support

• typing-extensions - Extended type hints

YouTube Features Status:

The following status information applies to YouTube transcript extraction:

• YouTube transcript extraction is stable and reliable with v1.1.0+

• No authentication or API keys required

• Works out of the box after installation

📄 License

MIT License

🤝 Contributing

This project implements the Model Context Protocol specification. It is compatible with any MCP-compliant client and built with the FastMCP framework for easy extension and modification.

📦 DXT Package Available

One-click installation for Claude Desktop users

This MCP server is available as a DXT (Desktop Extensions) package for easy installation. The following resources are available:

• DXT Package can be found at dxt-packages/crawl4ai-dxt-correct/

• Installation Guide is available at dxt-packages/README_DXT_PACKAGES.md

• Creation Guide is documented at dxt-packages/DXT_CREATION_GUIDE.md

• Troubleshooting information is at dxt-packages/DXT_TROUBLESHOOTING_GUIDE.md

Simply drag and drop the .dxt file into Claude Desktop for instant setup.

📚 Additional Documentation

Infrastructure & Deployment

• Docker Guide - Complete Docker containerization guide with multiple variants

• Architecture - Technical architecture, design decisions, and container infrastructure

• Build & Deployment - Build processes, CI/CD pipeline, and deployment strategies

• Configuration - Environment variables, Docker settings, and performance tuning

• Deployment Playbook - Production deployment procedures and troubleshooting

Development & Contributing

• Contributing Guide - Docker development workflow and contribution guidelines

• RunPod Deployment - Serverless cloud deployment on RunPod

• GitHub Actions - Automated CI/CD pipeline documentation

API & Integration

• Pure StreamableHTTP - Recommended HTTP transport protocol

• HTTP Server Usage - HTTP API server configuration

• Legacy HTTP API - Detailed HTTP API documentation

Localization & Support

• Japanese Documentation - Complete feature documentation in Japanese

• Japanese Troubleshooting - Troubleshooting guide in Japanese