Allows installation directly from GitHub repositories using the npx -y github:patrikmichi/crawlee-mcp command.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Crawlee MCP fetch the contents of https://example.com"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
A local Model Context Protocol server for web fetching, crawling, and screenshots with Crawlee. It runs over stdio, so it works with Claude Code, Claude Desktop, Codex, Cursor, and any MCP-compatible client.
Tool | Description |
| Fetch a single URL and return clean markdown |
| Fetch multiple URLs in parallel |
| Crawl a site, discover links, and return page content |
| Take a PNG screenshot of a page |
| Health check |
Related MCP server: Crawl4AI MCP
Requires Node.js 20 or newer. The first use of JavaScript rendering or screenshots also requires the Chromium browser:
npx playwright install chromiumOnce published to npm, every client can start it with:
npx -y @patrikmichi/crawlee-mcpUntil the npm package is published, install directly from GitHub:
npx -y github:patrikmichi/crawlee-mcpclaude mcp add --transport stdio crawlee -- npx -y @patrikmichi/crawlee-mcpAdd this to your Claude Desktop configuration file:
{
"mcpServers": {
"crawlee": {
"command": "npx",
"args": ["-y", "@patrikmichi/crawlee-mcp"]
}
}
}Add this to ~/.codex/config.toml:
[mcp_servers.crawlee]
command = "npx"
args = ["-y", "@patrikmichi/crawlee-mcp"]Use the same command and arguments: npx -y @patrikmichi/crawlee-mcp.
cp env.example .envVariable | Required | Description |
| No | Comma-separated proxy URLs for rotation |
| No | Request timeout in ms (default: |
| No | Set to |
npm install
npm start # Run with tsx (no build needed)
npm run build # Compile TypeScript
npm run typecheck # Type check onlyThis server is intentionally local and stdio-only because JavaScript rendering and screenshots require a local Playwright browser.
Crawl a website starting from a seed URL, following discovered links across multiple pages. Returns an array of {url, title, content} objects. Use link_pattern to restrict crawling to specific URL patterns (e.g. "https://example.com/blog/**").
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Starting URL to crawl from | |
| label | No | Label to attach to the initial request for categorising discovered links | |
| max_pages | No | Maximum number of pages to crawl. Default: 10, max: 50 | |
| render_js | No | Whether to use Playwright (JS rendering) or Cheerio (fast HTML-only). Default: true | |
| link_pattern | No | Glob pattern to filter discovered URLs (e.g. "https://example.com/blog/**"). Only URLs matching this pattern will be enqueued. | |
| link_selector | No | CSS selector for links to follow. Default: "a" | a |
| extract_content | No | Strip navigation, footer, and ads — return main content only. Default: true |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It discloses basic behavior (recursive crawling, output format) and hints at content extraction via extract_content. However, it omits important traits such as domain scope restrictions, robots.txt compliance, rate limiting, timeout behavior, and handling of external links. The description provides minimal behavioral context beyond what is already implied.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences long, well-structured with the main action first, then output format, then a focused usage hint. No fluff or redundancy; every sentence serves a purpose. It is front-loaded and easy to scan.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (7 parameters, no output schema) and the fact that schema coverage is 100%, the description covers the core purpose and output structure. It also highlights the distinguishing link_pattern parameter. However, it could be more complete by noting key parameters like max_pages or render_js, which affect behavior significantly. Still, the combination of description and well-documented schema provides a fairly comprehensive understanding.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so baseline is 3. The description adds slight value by explaining the output format and suggesting link_pattern usage, but it does not add meaning beyond the schema's own parameter descriptions. Each parameter is already clearly described in the schema with defaults and constraints.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool crawls a website by starting from a seed URL and following links across multiple pages, distinguishing it from sibling tools like fetch_url (single page) and fetch_multiple (specific URLs). It also explicitly describes the output format as an array of {url, title, content} objects, which provides concrete purpose clarity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not provide any guidance on when to use this tool versus alternatives like fetch_url or fetch_multiple. It only mentions using link_pattern to restrict crawling, which is a parameter usage tip, not a usage guideline. There is no mention of prerequisites, typical use cases, or when to avoid this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fetch multiple URLs in parallel and return an array of {url, content, error} objects. Useful for bulk research or comparison tasks.
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | Array of URLs to fetch (max 20) | |
| render_js | No | Whether to render JavaScript before extracting content. Default: true |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must fully disclose behavioral traits. It mentions parallel execution and per-URL error handling via the return structure ({url, content, error}). However, it does not mention rate limits, timeouts, auth requirements, or what happens if all URLs fail. The provided information is adequate but not comprehensive.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences, front-loading the core action and return format, and then adding a usage hint. Every sentence provides value, and there is no redundancy or fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has 2 parameters, no output schema, and no nested objects. The description covers the core purpose, parallel execution, return structure, and a usage context. It is largely complete for a tool of this complexity, though it could mention result ordering or URL validation behavior.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% (both parameters have descriptions in the input schema). The tool description adds no additional parameter semantics beyond what the schema already provides, so the baseline score of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Fetch multiple URLs in parallel'), the resource (URLs), and the return format (array of objects). It distinguishes from sibling tools like fetch_url (single URL) and screenshot (visual capture) by specifying parallel bulk fetching.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description includes 'Useful for bulk research or comparison tasks,' which implies a usage context but does not explicitly state when to use this tool versus alternatives (e.g., fetch_url for single URLs, screenshot for images). No exclusions or prerequisites are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Fetch a web page and return its content as clean markdown. Use render_js=true for JavaScript-heavy pages (SPAs, Twitter/X, etc). Use render_js=false for fast HTML-only fetching.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to fetch | |
| render_js | No | Whether to render JavaScript before extracting content. Default: true | |
| extract_content | No | Strip navigation, footer, and ads — return main content only. Default: true |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must fully disclose behavior. It states the output is clean markdown and hints at content extraction, but fails to mention error handling, timeouts, rate limits, or what happens with inaccessible pages. This is a significant gap for a web-fetching tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three concise sentences, front-loaded with the primary purpose, with no wasted words. Every sentence provides useful information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is adequate for straightforward use cases but lacks details on failure modes, limits, or what constitutes 'clean markdown'. Given no output schema, more context on return value behavior would be helpful for completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so a baseline of 3 applies. The description adds value by explaining render_js with concrete examples (SPAs, Twitter/X) and clarifying the speed trade-off, which enhances understanding beyond the schema. The extract_content parameter is not elaborated, but overall improvement is notable.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool fetches a web page and returns clean markdown, with a specific verb and resource. It hints at differentiation from siblings by mentioning JavaScript-heavy pages, but does not explicitly distinguish from fetch_multiple or screenshot, leaving some ambiguity.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear guidance on when to set render_js=true vs false, which is useful for parameter choice. However, it does not address when to use fetch_url over its siblings (fetch_multiple, screenshot, ping, crawl), so cross-tool selection guidance is missing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Check that the crawlee MCP server is running.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It only states the purpose but does not disclose the return value (e.g., boolean, status message) or any side effects. As a simple ping, the agent needs to know what the response looks like to interpret the result.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, efficient sentence that conveys the tool's purpose without any wasted words. It is appropriately sized for a simple tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has zero parameters, no annotations, and no output schema, the description is incomplete. It does not describe the return value or any error conditions, leaving the agent without essential information to use the tool effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
There are no parameters, and schema description coverage is 100%, so the baseline score of 3 applies. The description adds no extra parameter information, which is acceptable given the absence of parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool checks that the crawlee MCP server is running. This is a specific verb-check and resource-server, and it distinguishes itself from sibling tools like fetch_url, screenshot, and crawl, which perform actual data retrieval or actions.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives. It does not specify its role as a health check before other operations, missing an opportunity to clarify its usage context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Take a screenshot of a web page and return it as a base64-encoded PNG image.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to screenshot | |
| full_page | No | Capture the full scrollable page instead of only the viewport. Default: false |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description bears full responsibility for behavioral disclosure. It only states the basic operation and return format, omitting important details such as JavaScript rendering behavior, timeout policies, image dimensions, privacy/security restrictions, or whether dynamic content is captured. This leaves an agent uninformed about key side effects and constraints.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, short sentence that directly conveys the core functionality. It is not verbose and efficiently communicates the purpose. However, it could incorporate minimal additional context (e.g., output format clarity) without losing conciseness, earning a 4 rather than a 5.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that there is no output schema, the description correctly mentions the return type (base64-encoded PNG). However, it lacks details such as image resolution limits, ability to capture dynamic content, error responses, or size constraints. For a simple 2-param tool, it is mostly adequate but not fully complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the baseline is 3. The description does not add any additional semantic value beyond the schema's parameter descriptions (e.g., it does not clarify that 'full_page' captures the entire scrollable height). The existing schema descriptions are sufficient for understanding the two parameters.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Take a screenshot'), the target ('a web page'), and the return format ('base64-encoded PNG image'). It effectively distinguishes from sibling tools (fetch_url, fetch_multiple, ping, crawl) which are about raw content retrieval or connectivity rather than visual capture.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not explicitly state when to use this tool versus alternatives (e.g., 'Use this instead of fetch_url when you need the rendered page as an image'). Usage context is implied by the contrast with sibling tool names, but no explicit when-to-use or when-not-to-use guidance is provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
v1.1.0crawlfetch_multiplefetch_urlpingscreenshotEach tool has a clearly distinct purpose: fetching a single page, fetching multiple pages in parallel, taking a screenshot, health checking, and crawling a website. There is no overlap or ambiguity between the tools.
All tool names follow a consistent verb-based pattern using lowercase and underscores where needed (e.g., fetch_url, fetch_multiple). No mixing of styles like camelCase or inconsistent verb forms.
With 5 tools, the server is well-scoped for web crawling and scraping. Each tool earns its place: single fetch, batch fetch, screenshot, full crawl, and a health check. The count is neither too thin nor excessive.
The tool set covers the core workflows of fetching pages and crawling sites, with parallel fetching and screenshot support. Minor gaps exist, such as advanced extraction capabilities or session management, but the surface is practical for typical use cases.
Unclaimed servers have limited discoverability.
If you are the server author, to access and configure the admin panel.
Screenshot any URL/HTML as PNG/JPEG/WebP, or read it as clean Markdown/text for LLMs.
Stealth web automation for AI agents. Login, signup, navigate, screenshot.
Stealth web automation for AI agents. Login, signup, navigate, screenshot.
Fetch pages as markdown, search web and news, extract structured data. For AI agents.
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/patrikmichi/crawlee-mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server
