crawl4ai-mcp
Enables AI agents in Cursor Composer to scrape single webpages and crawl websites with configurable depth and page limits.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@crawl4ai-mcp scrape the latest AI news from techcrunch.com"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Crawl4AI MCP Server
A Model Context Protocol (MCP) server implementation that integrates Crawl4AI with Cursor AI, providing web scraping and crawling capabilities as tools for LLMs in Cursor Composer's agent mode.
System Requirements
Python 3.10 or higher installed.
Related MCP server: crawl4ai-mcp
Current Features
Single page scraping
Website crawling
Installation
Basic setup instructions also available in the Official Docs for MCP Server QuickStart.
Set up your environment
First, let's install uv and set up our Python project and environment:
MacOS/Linux:
curl -LsSf https://astral.sh/uv/install.sh | shWindows:
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"Make sure to restart your terminal afterwards to ensure that the uv command gets picked up.
After that:
Clone the repository
Install dependencies using UV:
# Navigate to the crawl4ai-mcp directory
cd crawl4ai-mcp
# Install dependencies (Only first time)
uv venv
uv sync
# Activate the venv
source .venv/bin/activate
# Run the server
python main.pyAdd to Cursor's MCP Servers or Claude's MCP Servers
You may need to put the full path to the uv executable in the command field. You can get this by running which uv on MacOS/Linux or where uv on Windows.
{
"mcpServers": {
"Crawl4AI": {
"command": "uv",
"args": [
"--directory",
"/ABSOLUTE/PATH/TO/PARENT/FOLDER/crawl4ai-mcp",
"run",
"main.py"
]
}
}
}Tools Provided
This MCP server exposes the following tools to the LLM:
scrape_webpage(url: str)Description: Scrapes the content and metadata from a single webpage using Crawl4AI.
Parameters:
url(string, required): The URL of the webpage to scrape.
Returns: A list containing a
TextContentobject with the scraped content (primarily markdown) as JSON.
crawl_website(url: str, crawl_depth: int = 1, max_pages: int = 5)Description: Crawls a website starting from the given URL up to a specified depth and page limit using Crawl4AI.
Parameters:
url(string, required): The starting URL to crawl.crawl_depth(integer, optional, default: 1): The maximum depth to crawl relative to the starting URL.max_pages(integer, optional, default: 5): The maximum number of pages to scrape during the crawl.
Returns: A list containing a
TextContentobject with a JSON array of results for the crawled pages (including URL, success status, markdown content, or error).
Available Tools
2 toolscrawl_websiteB
Crawl a website starting from the given URL up to a specified depth and page limit.
Args: url: The starting URL to crawl. crawl_depth: The maximum depth to crawl relative to the starting URL (default: 1). max_pages: The maximum number of pages to scrape during the crawl (default: 5).
Returns: List containing TextContent with a JSON array of results for crawled pages.
Inputs: url (required), crawl_depth, max_pages.
scrape_webpageC
Scrape content and metadata from a single webpage using Crawl4AI.
Args: url: The URL of the webpage to scrape
Returns: List containing TextContent with the result as JSON.
Inputs: url (required).
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 2 tools
The two tools have clearly distinct purposes: crawl_website handles multi-page crawling with depth and page limits, while scrape_webpage focuses on single-page extraction. There is no overlap in functionality, making it easy for an agent to choose the right tool based on whether it needs to crawl multiple pages or scrape just one.
Both tools follow a consistent verb_noun pattern (crawl_website and scrape_webpage), using snake_case and descriptive action-object naming. This consistency makes the tool set predictable and easy to understand at a glance.
With only 2 tools, the server feels too thin for a web crawling/scraping domain. While the tools cover basic crawling and scraping, typical MCP servers in this domain would include additional utilities like filtering, parsing, or handling different content types, making this set appear incomplete and limited in scope.
The tool set is severely incomplete for web crawling and scraping. It lacks essential operations such as updating crawl parameters, deleting or managing crawl jobs, handling errors or retries, and processing extracted data (e.g., cleaning, summarizing). This forces agents into dead ends for common workflows beyond basic one-off tasks.
Maintenance
Related MCP Connectors
One MCP server for all web data. Scrape, search, crawl, and AI research in one install.
Hosted Web Crawler MCP server for Claude, ChatGPT and Cursor, with each action switched per account.
Zenrows MCP server: Fetch, Extract, Batch, and Browser Sessions for AI agents
Web scraping MCP server — scrape, extract structured data, screenshot any site with anti-bot bypass.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceJava implementation of MCP Server for Crawl4ai4MIT
- AlicenseNot gradedqualityDmaintenanceMCP server integrating Crawl4AI for universal web crawling and data extraction. Enables AI agents to crawl, extract markdown/HTML, take screenshots, generate PDFs, and execute JavaScript on web pages.101 npm7MIT
- AlicenseNot gradedqualityDmaintenanceA lightweight MCP server that exposes Crawl4AI web scraping and crawling capabilities as tools for AI agents, enabling single-page scraping and multi-page crawling with adaptive stopping.118MIT
- AlicenseAqualityBmaintenanceMCP server for web search and crawling, integrating SearXNG metasearch and Crawl4AI for privacy-respecting search and content extraction.3149 PyPI190MIT