Skip to main content
Glama

Crawlee MCP

一个用于网页抓取、爬取和截图的本地模型上下文协议服务器,基于Crawlee。它通过 stdio 运行,因此可与 Claude Code、Claude Desktop、Codex、Cursor 以及任何兼容 MCP 的客户端配合使用。

工具

工具

描述

fetch_url

获取单个 URL 并返回干净的 markdown

fetch_multiple

并行获取多个 URL

crawl

爬取网站、发现链接并返回页面内容

screenshot

对页面进行 PNG 截图

ping

健康检查

Related MCP server: weblens-mcp

安装

需要 Node.js 20 或更高版本。首次使用 JavaScript 渲染或截图还需要 Chromium 浏览器:

npx playwright install chromium

一旦发布到 npm,每个客户端都可以通过以下方式启动:

npx -y @patrikmichi/crawlee-mcp

在 npm 包发布之前,直接从 GitHub 安装:

npx -y github:patrikmichi/crawlee-mcp

客户端配置

Claude Code

claude mcp add --transport stdio crawlee -- npx -y @patrikmichi/crawlee-mcp

Claude Desktop

将以下内容添加到 Claude Desktop 配置文件:

{
  "mcpServers": {
    "crawlee": {
      "command": "npx",
      "args": ["-y", "@patrikmichi/crawlee-mcp"]
    }
  }
}

Codex

将以下内容添加到 ~/.codex/config.toml:

[mcp_servers.crawlee]
command = "npx"
args = ["-y", "@patrikmichi/crawlee-mcp"]

其他 MCP 客户端

使用相同的命令和参数:npx -y @patrikmichi/crawlee-mcp。

配置

cp env.example .env

变量

必需

描述

CRAWLEE_PROXY_URLS

否

用于轮换的逗号分隔代理 URL

CRAWLEE_TIMEOUT_MS

否

请求超时时间(毫秒)(默认:30000)

CRAWLEE_STEALTH

否

设置为 false 以禁用隐身模式(默认:true)

开发

npm install
npm start          # Run with tsx (no build needed)
npm run build      # Compile TypeScript
npm run typecheck  # Type check only

此服务器特意设计为仅限本地和 stdio,因为 JavaScript 渲染和截图需要本地 Playwright 浏览器。

Available Tools

5 tools
crawlA

Crawl a website starting from a seed URL, following discovered links across multiple pages. Returns an array of {url, title, content} objects. Use link_pattern to restrict crawling to specific URL patterns (e.g. "https://example.com/blog/**").

Inputs: url (required), label, max_pages, render_js, +3 more.

fetch_multipleA

Fetch multiple URLs in parallel and return an array of {url, content, error} objects. Useful for bulk research or comparison tasks.

Inputs: urls (required), render_js.

fetch_urlB

Fetch a web page and return its content as clean markdown. Use render_js=true for JavaScript-heavy pages (SPAs, Twitter/X, etc). Use render_js=false for fast HTML-only fetching.

Inputs: url (required), render_js, extract_content.

pingB

Check that the crawlee MCP server is running.

Inputs: no named parameters.

screenshotB

Take a screenshot of a web page and return it as a base64-encoded PNG image.

Inputs: url (required), full_page.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.8/5.0

    Scored across 5 tools

    Disambiguation5/5

    Each tool has a clearly distinct purpose: fetching a single page, fetching multiple pages in parallel, taking a screenshot, health checking, and crawling a website. There is no overlap or ambiguity between the tools.

    Naming Consistency5/5

    All tool names follow a consistent verb-based pattern using lowercase and underscores where needed (e.g., fetch_url, fetch_multiple). No mixing of styles like camelCase or inconsistent verb forms.

    Tool Count5/5

    With 5 tools, the server is well-scoped for web crawling and scraping. Each tool earns its place: single fetch, batch fetch, screenshot, full crawl, and a health check. The count is neither too thin nor excessive.

    Completeness4/5

    The tool set covers the core workflows of fetching pages and crawling sites, with parallel fetching and screenshot support. Minor gaps exist, such as advanced extraction capabilities or session management, but the surface is practical for typical use cases.

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables web scraping and crawling capabilities for LLM clients, supporting single-page scraping, multi-page website crawling, and web search with multiple engines (Playwright, Cheerio, Puppeteer) and flexible output formats including markdown, HTML, text, and screenshots.
      7
      MIT
    • A
      license
      A
      quality
      D
      maintenance
      Enables AI agents to fetch and render web pages (including JavaScript-heavy SPAs) with headless Chromium, extract readable content with Mozilla Readability, capture navigation links, download images, and return a clean markdown file path.
      1
      17 npm
      ISC