Crawl4AI MCP Server

A powerful Model Context Protocol (MCP) server that provides web scraping and crawling capabilities using Crawl4AI. This server acts as the "hands and eyes" for client-side AI, enabling intelligent web content analysis and extraction.

Features

• 🔍 Page Structure Analysis: Extract clean HTML or Markdown content from any webpage

• 🎯 Schema-Based Extraction: Precision data extraction using CSS selectors and AI-generated schemas

• 📸 Screenshot Capture: Visual webpage representation for analysis

• ⚡ Async Operations: Non-blocking web crawling with progress reporting

• 🛡️ Error Handling: Comprehensive error handling and validation

• 📊 MCP Integration: Full Model Context Protocol compatibility with logging and progress tracking

Architecture

• FastMCP: Handles MCP protocol and tool registration

• AsyncWebCrawler: Provides async web scraping capabilities

• Stdio Transport: MCP-compatible communication channel

• Error-Safe Logging: All logs directed to stderr to prevent protocol corruption

Installation

Prerequisites

• Python 3.10 or higher

• pip package manager

Setup

1. Clone or download this repository:

   git clone <repository-url> cd crawl4ai-mcp

2. Create and activate virtual environment:

   python3 -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate

3. Install dependencies:

   pip install -r requirements.txt

4. Install Playwright browsers (required for screenshots):

   playwright install

Usage

Starting the Server

Testing with MCP Inspector

For interactive testing and development:

This will start a web interface (usually at http://localhost:6274) where you can test all tools interactively.

Available Tools

1. server_status

Purpose: Get server health and capabilities information
Parameters: None

Example Response:

2. get_page_structure

Purpose: Extract webpage content for analysis (the "eyes" function)
Parameters:

• url (string): The webpage URL to analyze

• format (string, optional): Output format - "html" or "markdown" (default: "html")

Example:

3. crawl_with_schema

Purpose: Precision data extraction using CSS selectors (the "hands" function)
Parameters:

• url (string): The webpage URL to extract data from

• extraction_schema (string): JSON string defining field names and CSS selectors

Example Schema:

Example Usage:

4. take_screenshot

Purpose: Capture visual representation of webpage
Parameters:

• url (string): The webpage URL to screenshot

Example:

Returns: Base64-encoded PNG image data with metadata

Integration with Claude Desktop

To use this server with Claude Desktop, add this configuration to your Claude Desktop settings:

Replace /path/to/crawl4ai-mcp/ with the actual path to your installation directory.

Error Handling

All tools include comprehensive error handling and return structured JSON responses:

Common error scenarios:

• Invalid URL format

• Network connectivity issues

• Invalid extraction schemas

• Screenshot capture failures

Development

Project Structure

Dependencies

• fastmcp: FastMCP framework for MCP server development

• crawl4ai: Core web crawling and extraction library

• pydantic: Data validation and parsing

• playwright: Browser automation for screenshots

Testing

Run the linter to ensure code quality:

Test server startup:

Contributing

1. Fork the repository

2. Create a feature branch

3. Make your changes

4. Test thoroughly with MCP Inspector

5. Submit a pull request

License

This project is open source. See the LICENSE file for details.

Support

For issues and questions:

1. Check the troubleshooting section in USAGE_EXAMPLES.md

2. Test with MCP Inspector to isolate issues

3. Verify all dependencies are correctly installed

4. Ensure virtual environment is activated

Acknowledgments

• Crawl4AI: Powerful web crawling and extraction capabilities

• FastMCP: Streamlined MCP server development framework

• Model Context Protocol: Standardized AI tool integration