Crawl4AI MCP Server

# Crawl4AI MCP Server A powerful Model Context Protocol (MCP) server that provides web scraping and crawling capabilities using Crawl4AI. This server acts as the "hands and eyes" for client-side AI, enabling intelligent web content analysis and extraction. ## Features - **🔍 Page Structure Analysis**: Extract clean HTML or Markdown content from any webpage - **🎯 Schema-Based Extraction**: Precision data extraction using CSS selectors and AI-generated schemas - **📸 Screenshot Capture**: Visual webpage representation for analysis - **⚡ Async Operations**: Non-blocking web crawling with progress reporting - **🛡️ Error Handling**: Comprehensive error handling and validation - **📊 MCP Integration**: Full Model Context Protocol compatibility with logging and progress tracking ## Architecture ``` ┌─────────────────┐ ┌───────────────────┐ ┌─────────────────┐ │ Client AI │ │ Crawl4AI MCP │ │ Web Content │ │ ("Brain") │◄──►│ Server │◄──►│ (Websites) │ │ │ │ ("Hands & Eyes") │ │ │ └─────────────────┘ └───────────────────┘ └─────────────────┘ ``` - **FastMCP**: Handles MCP protocol and tool registration - **AsyncWebCrawler**: Provides async web scraping capabilities - **Stdio Transport**: MCP-compatible communication channel - **Error-Safe Logging**: All logs directed to stderr to prevent protocol corruption ## Installation ### Prerequisites - Python 3.10 or higher - pip package manager ### Setup 1. **Clone or download this repository**: ```bash git clone <repository-url> cd crawl4ai-mcp ``` 2. **Create and activate virtual environment**: ```bash python3 -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate ``` 3. **Install dependencies**: ```bash pip install -r requirements.txt ``` 4. **Install Playwright browsers** (required for screenshots): ```bash playwright install ``` ## Usage ### Starting the Server ```bash # Activate virtual environment source venv/bin/activate # Start the MCP server python3 crawl4ai_mcp_server.py ``` ### Testing with MCP Inspector For interactive testing and development: ```bash # Start MCP Inspector interface fastmcp dev crawl4ai_mcp_server.py ``` This will start a web interface (usually at http://localhost:6274) where you can test all tools interactively. ## Available Tools ### 1. server_status **Purpose**: Get server health and capabilities information **Parameters**: None **Example Response**: ```json { "server_name": "Crawl4AI-MCP-Server", "version": "1.0.0", "status": "operational", "capabilities": ["web_crawling", "content_extraction", "screenshot_capture", "schema_based_extraction"] } ``` ### 2. get_page_structure **Purpose**: Extract webpage content for analysis (the "eyes" function) **Parameters**: - `url` (string): The webpage URL to analyze - `format` (string, optional): Output format - "html" or "markdown" (default: "html") **Example**: ```json { "url": "https://example.com", "format": "html" } ``` ### 3. crawl_with_schema **Purpose**: Precision data extraction using CSS selectors (the "hands" function) **Parameters**: - `url` (string): The webpage URL to extract data from - `extraction_schema` (string): JSON string defining field names and CSS selectors **Example Schema**: ```json { "title": "h1", "description": "p.description", "price": ".price-value", "author": ".author-name", "tags": ".tag" } ``` **Example Usage**: ```json { "url": "https://example.com/product", "extraction_schema": "{\"title\": \"h1\", \"price\": \".price\", \"description\": \"p\"}" } ``` ### 4. take_screenshot **Purpose**: Capture visual representation of webpage **Parameters**: - `url` (string): The webpage URL to screenshot **Example**: ```json { "url": "https://example.com" } ``` **Returns**: Base64-encoded PNG image data with metadata ## Integration with Claude Desktop To use this server with Claude Desktop, add this configuration to your Claude Desktop settings: ```json { "mcpServers": { "crawl4ai": { "command": "python3", "args": ["/path/to/crawl4ai-mcp/crawl4ai_mcp_server.py"], "env": {} } } } ``` Replace `/path/to/crawl4ai-mcp/` with the actual path to your installation directory. ## Error Handling All tools include comprehensive error handling and return structured JSON responses: ```json { "error": "Error description", "url": "https://example.com", "success": false } ``` Common error scenarios: - Invalid URL format - Network connectivity issues - Invalid extraction schemas - Screenshot capture failures ## Development ### Project Structure ``` crawl4ai-mcp/ ├── crawl4ai_mcp_server.py # Main server implementation ├── requirements.txt # Python dependencies ├── pyproject.toml # Project configuration ├── USAGE_EXAMPLES.md # Detailed usage examples └── README.md # This file ``` ### Dependencies - **fastmcp**: FastMCP framework for MCP server development - **crawl4ai**: Core web crawling and extraction library - **pydantic**: Data validation and parsing - **playwright**: Browser automation for screenshots ### Testing Run the linter to ensure code quality: ```bash ruff check . ``` Test server startup: ```bash python3 crawl4ai_mcp_server.py ``` ## Contributing 1. Fork the repository 2. Create a feature branch 3. Make your changes 4. Test thoroughly with MCP Inspector 5. Submit a pull request ## License This project is open source. See the LICENSE file for details. ## Support For issues and questions: 1. Check the troubleshooting section in USAGE_EXAMPLES.md 2. Test with MCP Inspector to isolate issues 3. Verify all dependencies are correctly installed 4. Ensure virtual environment is activated ## Acknowledgments - **Crawl4AI**: Powerful web crawling and extraction capabilities - **FastMCP**: Streamlined MCP server development framework - **Model Context Protocol**: Standardized AI tool integration