Crawlee MCP
Allows installation directly from GitHub repositories using the npx -y github:patrikmichi/crawlee-mcp command.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Crawlee MCP fetch the contents of https://example.com"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Crawlee MCP
A local Model Context Protocol server for web fetching, crawling, and screenshots with Crawlee. It runs over stdio, so it works with Claude Code, Claude Desktop, Codex, Cursor, and any MCP-compatible client.
Tools
Tool | Description |
| Fetch a single URL and return clean markdown |
| Fetch multiple URLs in parallel |
| Crawl a site, discover links, and return page content |
| Take a PNG screenshot of a page |
| Health check |
Related MCP server: weblens-mcp
Install
Requires Node.js 20 or newer. The first use of JavaScript rendering or screenshots also requires the Chromium browser:
npx playwright install chromiumOnce published to npm, every client can start it with:
npx -y @patrikmichi/crawlee-mcpUntil the npm package is published, install directly from GitHub:
npx -y github:patrikmichi/crawlee-mcpClient configuration
Claude Code
claude mcp add --transport stdio crawlee -- npx -y @patrikmichi/crawlee-mcpClaude Desktop
Add this to your Claude Desktop configuration file:
{
"mcpServers": {
"crawlee": {
"command": "npx",
"args": ["-y", "@patrikmichi/crawlee-mcp"]
}
}
}Codex
Add this to ~/.codex/config.toml:
[mcp_servers.crawlee]
command = "npx"
args = ["-y", "@patrikmichi/crawlee-mcp"]Other MCP clients
Use the same command and arguments: npx -y @patrikmichi/crawlee-mcp.
Configuration
cp env.example .envVariable | Required | Description |
| No | Comma-separated proxy URLs for rotation |
| No | Request timeout in ms (default: |
| No | Set to |
Development
npm install
npm start # Run with tsx (no build needed)
npm run build # Compile TypeScript
npm run typecheck # Type check onlyThis server is intentionally local and stdio-only because JavaScript rendering and screenshots require a local Playwright browser.
Available Tools
5 toolscrawlA
Crawl a website starting from a seed URL, following discovered links across multiple pages. Returns an array of {url, title, content} objects. Use link_pattern to restrict crawling to specific URL patterns (e.g. "https://example.com/blog/**").
Inputs: url (required), label, max_pages, render_js, +3 more.
fetch_multipleA
Fetch multiple URLs in parallel and return an array of {url, content, error} objects. Useful for bulk research or comparison tasks.
Inputs: urls (required), render_js.
fetch_urlB
Fetch a web page and return its content as clean markdown. Use render_js=true for JavaScript-heavy pages (SPAs, Twitter/X, etc). Use render_js=false for fast HTML-only fetching.
Inputs: url (required), render_js, extract_content.
pingB
Check that the crawlee MCP server is running.
Inputs: no named parameters.
screenshotB
Take a screenshot of a web page and return it as a base64-encoded PNG image.
Inputs: url (required), full_page.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 5 tools
Each tool has a clearly distinct purpose: fetching a single page, fetching multiple pages in parallel, taking a screenshot, health checking, and crawling a website. There is no overlap or ambiguity between the tools.
All tool names follow a consistent verb-based pattern using lowercase and underscores where needed (e.g., fetch_url, fetch_multiple). No mixing of styles like camelCase or inconsistent verb forms.
With 5 tools, the server is well-scoped for web crawling and scraping. Each tool earns its place: single fetch, batch fetch, screenshot, full crawl, and a health check. The count is neither too thin nor excessive.
The tool set covers the core workflows of fetching pages and crawling sites, with parallel fetching and screenshot support. Minor gaps exist, such as advanced extraction capabilities or session management, but the surface is practical for typical use cases.
Related MCP Connectors
Fetch any URL as clean markdown, HTML, or a screenshot for AI agents.
Web scraping, browser automation, screenshots and AI data extraction for AI agents via MCP.
Web data for agents. Extract web pages, crawl websites, discover URLs, and search the web.
Screenshot any URL/HTML as PNG/JPEG/WebP, or read it as clean Markdown/text for LLMs.
Related MCP Servers
- AlicenseNot gradedqualityCmaintenanceEnables web scraping and crawling capabilities for LLM clients, supporting single-page scraping, multi-page website crawling, and web search with multiple engines (Playwright, Cheerio, Puppeteer) and flexible output formats including markdown, HTML, text, and screenshots.7MIT
- AlicenseAqualityDmaintenanceEnables AI agents to fetch and render web pages (including JavaScript-heavy SPAs) with headless Chromium, extract readable content with Mozilla Readability, capture navigation links, download images, and return a clean markdown file path.117 npmISC
- AlicenseAqualityDmaintenanceEnables AI agents to fetch any web page as clean markdown or screenshot it, turning URLs into LLM-ready context.24 npmMIT
- AlicenseAqualityBmaintenanceEnables AI agents to capture screenshots, scrape structured page content, and run automated visual checks on live websites via a headless Chromium browser.3MIT