Summary: A local stdio MCP server that crawls web pages with Crawl4ai and returns Markdown, structured JSON, or full-page screenshots.
crawl_markdown(url, wait_seconds=0, wait_selector="") — Crawl a URL and get its content as Markdown.
crawl_structured(url, selector, fields, wait_seconds=0, wait_selector="") — Extract repeated elements into JSON via CSS selectors, with per-field specs for text (td / td:text), child attributes (a@href), base-element attributes (@data-value), and nth-element selection (td:nth-of-type(1); :eq() unsupported).
crawl_screenshot(url, output_path) — Capture a full-page screenshot to a file (parent dirs auto-created); Windows paths are converted to /mnt/<drive>/..., and system directories are blocked for safety.
Dynamic page support — All crawl tools accept wait_seconds (fixed delay) or wait_selector (wait for a CSS selector, takes precedence).
Reliable browsing — A single Chromium instance is reused across calls, with automatic crash detection and retry (up to 2 recreations) and safe concurrent crawling.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@Crawl4ai Local Extract book titles and prices from https://books.toscrape.com"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Local MCP server for Crawl4ai on Linux.
🤖 Most of the code was generated with AI assistance.
See also:
Linux
Python 3.14 (installing via uv is fine: uv python install 3.14)
Related MCP server: agent-browser
uv sync
# Install Chromium for Playwright
uv run crawl4ai-setupIf the browser fails to launch because system libraries required by Chromium are missing, install them with the command below (requires sudo; apt-based distros such as Debian/Ubuntu only).
uv run playwright install-deps chromium🚨 This server is a stdio server — no need to keep it running like a network server. When connected to an agent as MCP, the agent automatically spins up the server instance.
# Check for the "🚀 Crawl4ai MCP server started." message, then exit
uv run main✅ After confirming the welcome message, press Ctrl+C to exit, then add the MCP to your agent.
# Tip: debugging mode (MCP Inspector)
uv run mcp dev src/crawl4ai_local/server.pyclaude mcp add --transport stdio --scope user crawl4ai -- uv run --directory <parent_location>/crawl4ai-local mainparent_location: An absolute path like /home/<user>/dev/repo (use ~ only where the shell expands it)
e.g. claude mcp add --transport stdio --scope user crawl4ai -- uv run --directory /home/me/dev/repo/crawl4ai-local main
# Check MCP installation
claude mcp list
claude mcp get crawl4aihermes mcp add crawl4ai --command "uv" --args "run" "--directory" "<parent_location>/crawl4ai-local" "main"parent_location: An absolute path like /home/<user>/dev/repo (use ~ only where the shell expands it)
e.g. /home/me/dev/repo/crawl4ai-local
# Check MCP installation
hermes mcp listcrawl_markdown(url, wait_seconds=0, wait_selector="")Crawl a URL and return the content as Markdown.
crawl_structured(url, selector, fields, wait_seconds=0, wait_selector="")Extract repeated elements as JSON using CSS selectors.
crawl_structured(
url="https://books.toscrape.com/",
selector="article.product_pod",
fields={"제목": "h3 a@title", "가격": ".price_color:text"},
)
# → [{"제목": "A Light in the Attic", "가격": "£51.77"}, ...]Field spec syntax:
Spec | Meaning |
| Text of the matched element |
| Attribute of a child element ( |
| Attribute of the base element itself |
| Nth element — use standard CSS ( |
Both tools share the waiting options (see below).
crawl_screenshot(url, output_path)Capture a full-page screenshot and save it to a file. Parent directories are created automatically.
💡 If output_path is a Windows path (C:\Users\me\shot.png, D:/shots/a.png), it is converted to the mounted drive path (/mnt/c/Users/me/shot.png) before saving.
🛡️ output_path is checked against a blocklist of known system directories before the crawl even runs.
Linux: /bin, /boot, /dev, /etc, /lib, /lib32, /lib64, /libx32, /proc, /run, /sbin, /snap, /sys, /usr, /var
Windows (under /mnt/<drive>/, case-insensitive): Windows, Program Files, Program Files (x86), ProgramData, System Volume Information, $Recycle.Bin, Users/All Users, Users/Default
A path that resolves into one of these (including via .. traversal or symlinks) is rejected with an error message, so an agent can't accidentally overwrite system files. This is a blocklist, not a full sandbox — it guards against mistakes, not a determined attacker. If Windows drives are mounted somewhere other than /mnt/, update _DRIVE_MOUNT_ROOT in server.py accordingly.
Instead of launching Chromium on every call (which is expensive), the server keeps one browser instance for the process lifetime. If the shared browser crashes mid-session, the server detects it and recovers automatically:
Crash detection matches Playwright collapse signatures only — Target page, context or browser has been closed, browser has crashed, browsertype.launch failures, etc. Network errors, anti-bot blocks, and timeouts are page-side causes and are never retried with a fresh browser.
On crash: dispose the dead instance → launch a new one → retry, up to _MAX_RECREATE_ATTEMPTS = 2 attempts.
Both failure paths are checked equally on every attempt: exceptions raised by arun() and result.success=False + crash message in error_message.
On the last attempt, a crash is still reported as a failure, but the crawler is not reset again — that cleanup already happened on the prior attempt, and resetting a second time could tear down an instance a concurrent request just recreated.
Preventive recycling (e.g., refresh every N pages) is intentionally left to Crawl4ai's built-in browser recycling; the server only reacts to actual collapses.
Concurrency note: multiple simultaneous arun() calls on the shared crawler are safe — Crawl4ai serializes page creation internally (_page_lock, GH-1198 fix) and manages context lifecycle with refcounting + LRU. Verified with concurrent multi-site smoke tests.
For dynamic pages that render content late using JS, use the waiting options of crawl_markdown / crawl_structured.
# Method 1: Wait for a fixed duration (seconds)
crawl_markdown(url="https://example.com", wait_seconds=5)
# Method 2: Wait until a CSS selector appears (takes precedence over wait_seconds when specified)
crawl_markdown(url="https://example.com", wait_selector="div.result-list")Comparison of 3 cases on a local test page that updates content via JS after 3 seconds:
Case | success | Captures Dynamic Content |
No waiting option | True | ❌ Returns only "Loading..." |
| True | ✅ Accurately captures final content |
| True | ❌ Passes immediately if the element already exists in the initial HTML |
wait_selector is only valid for "newly created elements." If the text of an element already present in the initial HTML (e.g., <article id="dynamic">Loading...</article>) changes later, the selector matches immediately and passes without waiting. For pages that update text dynamically, use wait_seconds.
Extreme pages trigger anti-bot heuristics. Pages with minimal content and many script tags are flagged by Crawl4ai's anti-bot detector as Blocked by anti-bot protection: Structural: no_content_elements, script_heavy_shell, resulting in success=False. Although rare in production pages, testing should be done on pages containing basic static content (navigation bars, paragraphs, etc.).
This server depends on the official crawl4ai library, so upstream releases may require source changes. The project ships a Claude Code slash command that handles this: .claude/commands/sync-upstream.md.
Run it in Claude Code from this repository's root:
# Sync to the latest stable release
/sync-upstream
# Sync to a specific version (pre-releases only when named explicitly)
/sync-upstream 0.9.5What it does:
Compares the installed crawl4ai version with the latest GitHub release, and stops if already up to date.
Reads the release notes and CHANGELOG.md since the current version, keeping only changes that affect the crawl4ai APIs used in src/.
Raises the crawl4ai>= lower bound with uv add, and runs uv run crawl4ai-setup when the browser version changed.
Updates server.py (and both READMEs when behavior changes) without changing MCP tool names, arguments, or return formats. If that becomes unavoidable, it asks first.
Runs Ruff, Flake8, mypy, pytest, and a live smoke test of all three tools, then reports the results.
It does not commit. Review the diff and commit yourself.
주어진 URL을 크롤링해서 마크다운으로 반환한다.
Args: url: 크롤링할 URL. wait_seconds: HTML을 받기 전 대기 시간(초). JS 동적 로딩 페이지에 사용. wait_selector: 이 CSS 셀렉터가 나타날 때까지 대기. 지정 시 wait_seconds보다 우선.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| wait_seconds | No | ||
| wait_selector | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden of behavioral disclosure. It does mention the waiting behavior for dynamic pages (wait_seconds, wait_selector) and the priority of wait_selector, which is useful. However, it omits error handling, timeout behavior, authentication requirements, or any side effects beyond fetching the page.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and well-structured, with a one-sentence summary followed by a clear parameter list. It is front-loaded with the core function and avoids unnecessary detail. The only minor loss is not using bullet points, but the format is acceptable.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the tool's purpose and parameters adequately, and an output schema exists to define the return format. However, it does not mention any limitations, error scenarios, or when it should not be used, and it lacks explicit differentiation from siblings. For a tool with three parameters and no annotations, it is slightly below the ideal completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description includes an Args section that explains each parameter beyond the schema titles. It clarifies that wait_seconds is for JS dynamic loading and that wait_selector overrides wait_seconds when specified. This compensates for the 0% schema description coverage and adds meaningful semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (crawl), the target (URL), and the output format (markdown). It is specific enough to distinguish from the sibling tools (crawl_screenshot and crawl_structured) based on the output type, though it does not explicitly name the alternatives.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no guidance on when to use this tool versus the siblings, nor any exclusions or alternative routing. Usage context is only implied by the markdown output, which is insufficient for an agent to make a confident choice.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
스크린샷을 찍어 파일로 저장한다.
Args: url: 크롤링할 URL. output_path: 저장할 파일 경로. Windows 경로("C:\Users\me\x.png")는 /mnt/c/Users/me/x.png 로 변환해 저장한다.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| output_path | Yes |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses one behavioral trait: Windows paths are converted to /mnt/c/ format. However, it does not mention other relevant behaviors such as network access, headless browsing, whether the page must be fully loaded, or potential side effects like cookie usage. Given the tool's simplicity, the path conversion is helpful but insufficient for full transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise: one purpose sentence plus a formatted Args block. The key behavioral note about path conversion is included without redundancy. The structure is clean and front-loaded, with the essential purpose stated first and parameter details following. Every sentence earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a two-parameter screenshot tool, the description covers the basic usage and the path conversion quirk. However, it omits any mention of return values (though an output schema exists), error conditions, or runtime requirements (e.g., network access, browser dependencies). It is adequate for simple usage but leaves gaps that could cause the agent to call it incorrectly in edge cases.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate. For 'url', it merely says 'URL to crawl,' which adds little beyond the parameter name. For 'output_path', it explains the Windows-to-WSL path conversion, which is genuinely useful and goes beyond the schema. This partial enrichment earns a 3 – it provides some semantic value but is not comprehensive.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Take a screenshot and save it as a file.' It specifies the resource (screenshot) and the action (save to file), which is specific enough. It does not explicitly differentiate from sibling tools like crawl_markdown or crawl_structured, but the name and action make it obvious it's for capturing screenshots rather than extracting text or structured data, so it's clear but not explicitly contrasted.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this tool versus the siblings. It does not mention any alternative or exclusion criteria. The only usage-related note is about Windows path conversion, which is a technical detail rather than a decision guideline. The agent is left to infer that this tool is appropriate when a screenshot is needed, but no explicit when-to-use or when-not-to-use information is provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
CSS 셀렉터로 반복 요소를 잡아 지정한 필드만 JSON으로 추출한다.
Args: url: 크롤링할 URL. selector: 반복 요소를 잡는 CSS 셀렉터 (예: "table tr.item", "div.product-card"). fields: 추출할 필드. 키=필드명, 값=추출 지정 문자열. - 텍스트: "td" 또는 "td:text" - 속성: "a@href" (요소@속성명), 요소 자체 속성은 "@data-value" - N번째 요소는 CSS 문법 "td:nth-of-type(1)" 사용 (":eq()" 미지원) 예: {"이름": "td:nth-of-type(1):text", "링크": "a@href"} wait_seconds: HTML을 받기 전 대기 시간(초). JS 동적 로딩 페이지에 사용. wait_selector: 이 CSS 셀렉터가 나타날 때까지 대기. 지정 시 wait_seconds보다 우선.
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| fields | Yes | ||
| selector | Yes | ||
| wait_seconds | No | ||
| wait_selector | No |
| Name | Required | Description |
|---|---|---|
| result | Yes |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the behavioral disclosure burden. It discloses that wait_selector takes precedence over wait_seconds, that :eq() is not supported, and how attribute extraction works. It does not cover edge cases like missing elements or error behavior, but the output schema presumably handles return-value details.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The one-sentence purpose is front-loaded, followed by a well-organized Args block where each line contributes actionable detail. The syntax examples and precedence note are dense but avoid redundant or promotional language. Nothing in the description is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers all five parameters, including the important wait_selector precedence and field extraction grammar, which is sufficient for invoking the tool. Given that an output schema exists, return values need not be described. Minor gaps such as behavior on empty selections or pagination prevent a 5.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description fully compensates. Every parameter is documented with concrete examples: url, selector with illustrative patterns, fields with text/attribute/Nth-child syntax and a JSON example, plus wait_seconds and wait_selector with usage guidance. This goes far beyond the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: using CSS selectors to capture repeating elements and extract only designated fields into JSON. This clearly distinguishes it from sibling tools crawl_markdown and crawl_screenshot, which produce different output formats.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context on how to use the tool, including examples of selectors and field extraction syntax, and notes that wait_seconds is for JS-dynamic pages. However, it does not explicitly state when to prefer this over crawl_markdown or crawl_screenshot, so it lacks explicit exclusions and alternative routing.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0crawl_markdowncrawl_screenshotcrawl_structuredScored across 3 tools
Each tool converts a URL into a different output format: markdown, screenshot, or structured JSON. The purposes are mutually exclusive and clearly distinguishable by the suffix in the tool name. No overlap or ambiguity exists.
All tools share the consistent 'crawl_' prefix followed by the output type (markdown, screenshot, structured). This follows a predictable verb_noun pattern, making it easy to infer tool behavior from the name alone.
Three tools is well-scoped for a focused crawling server that offers three distinct output modes. Each tool earns its place without redundancy or unnecessary bloat.
The set covers the core crawling needs: full content (markdown), visual capture (screenshot), and targeted extraction (structured). A minor gap is the lack of a raw HTML option, but the three provided modes handle most practical use cases without dead ends.
Web scraping, browser automation, screenshots and AI data extraction for AI agents via MCP.
Scrape, crawl and search the web for AI agents via MCP.
Web scraping MCP server: any URL to Markdown or JSON, with JavaScript rendering and batch jobs.
Stealth web browser for agents: search, fetch, click, download and type in persistent MCP sessions.
