사이트 전체를 크롤링하는 MCP 서버로, crawl4ai(Apache-2.0) 기반입니다. 검색 엔진 스파이더처럼 웹사이트의 모든 내부 링크를 크롤링하고(BFS), 게시 시간/제목/URL로 페이지를 필터링하며, 크롤링된 페이지에서 작업(요약, 링크 클릭, 파일 다운로드)을 수행할 수 있습니다. 결과는 JSON으로 반환되고 SQLite에 저장되어 어떤 프로젝트에서도 재사용할 수 있습니다.
한국어 소개: 「구글 크롤러처럼」 사이트 전체 내부 링크를 크롤링하는 MCP입니다. crawl4ai 기반으로 자체 래핑했으며, BFS 전체 사이트 탐색, 시간/제목/URL 필터링, 페이지 작업(LLM 요약/링크 클릭/파일 다운로드), 결과 JSON 반환 + SQLite 영속화를 지원합니다. Reasonix / Claude Desktop / Cursor 등 어떤 MCP 클라이언트에든 등록하면 전역에서 재사용할 수 있습니다.
✅ 사이트 전체 BFS 크롤링 — 모든 내부 링크를 순회합니다 (max_depth / max_pages 조절 가능)
✅ 시간 필터 — 페이지 메타 / JSON-LD / URL에서 게시 시간을 먼저 추출하고,
없는 경우 크롤링 시간으로 대체합니다 (결과는 time_source로 표시)
✅ 제목 필터 — 제목 키워드로 포함/제외 (대소문자 구분 안 함)
✅ URL 패턴 및 도메인 필터 — glob/regex URL 매칭, 동일 도메인 제한
✅ 정중한 크롤링 — 기본적으로 robots.txt를 존중하고 속도 제한을 적용합니다 (전환 가능)
✅ 페이지 작업 — LLM 요약 (LiteLLM: DeepSeek / GLM / OpenAI…, 로컬 폴백), 특정 링크 클릭 (CSS 선택자 또는 링크 텍스트), 파일 다운로드
✅ SQLite 영속화 — 나중 프로젝트에서 크롤링 데이터를 재사용할 수 있는 query_crawls 도구
Related MCP server: Spider MCP Server
도구 | 설명 |
| BFS 전체 사이트 크롤링. 매개변수: |
| 단일 페이지 스크레이핑 (markdown / 제목 / 게시 시간 / 링크) |
| 페이지 콘텐츠 요약. |
| 페이지 내부의 링크 클릭 ( |
| 페이지 파일을 출력 디렉터리로 다운로드 (기본값 |
| SQLite에서 저장된 크롤링 결과 쿼리 (제목/URL/시간 필터) |
Python ≥ 3.12 (3.12.13에서 테스트됨)
uv 권장 (선택 사항 — 일반 pip도 작동합니다)
Playwright 브라우저: python -m playwright install chromium
(또는 PLAYWRIGHT_BROWSERS_PATH를 기존 브라우저 설치 경로로 설정)
# 1. Create environment & install
uv venv .venv --python 3.12
uv pip install --python .venv\Scripts\python.exe crawl4ai "mcp>=1.2,<2"
uv pip install --python .venv\Scripts\python.exe -e .
# 2. Install browser (once)
.venv\Scripts\python.exe -m playwright install chromium
# 3. Register as MCP server (example for Reasonix config.toml)
[[plugins]]
name = "site-crawler-mcp"
type = "stdio"
command = "C:\\path\\to\\site-crawler-mcp\\.venv\\Scripts\\python.exe"
args = ["-m", "site_crawler_mcp.server"]Claude Desktop / Cursor의 경우, 해당 설정 파일의 mcpServers 아래에 동일한 command/args를 추가하세요.
import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
async def main():
params = StdioServerParameters(command="python", args=["-m", "site_crawler_mcp.server"])
async with stdio_client(params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
res = await session.call_tool("crawl_site", {
"start_url": "https://example.com",
"max_depth": 2,
"max_pages": 20,
"title_contains": "Example",
})
print(res.content[0].text)
asyncio.run(main())crawl4ai의 URL 필터는 URL에만 적용되므로, 여기서는 콘텐츠 수준 필터링을 구현했습니다:
URL 수준 가지치기 — TimeRangeFilter / TitleFilter (URL 날짜 패턴, URL 키워드)
콘텐츠 수준 — 각 페이지를 가져온 후 <meta property="article:published_time">, JSON-LD datePublished, <time datetime>, URL의 YYYY/MM/DD에서 게시 시간을 추출합니다. 찾지 못하면 크롤링 시간을 사용합니다 (결정 사항은 time_source로 기록됨).
기본적으로 robots.txt와 속도 제한을 존중하여 대상 사이트에 부담을 주거나 IP가 차단되는 것을 방지합니다.
학습 / 연구 / 자신의 사이트를 위한 용도입니다. 대상 사이트의 이용약관과 현지 법률을 준수하세요.
crawl4ai(Apache-2.0) 기반으로 제작되었습니다.
点击页面内指定链接（CSS selector 或链接文本）并抓取目标页内容。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| selector | No | ||
| link_text | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden for behavioral disclosure. It mentions 'click and scrape' but omits critical traits: what happens if the link is not found, whether JavaScript-triggered navigation is supported, whether it waits for page load, or if it modifies browser state. The description is too minimal to convey the tool's operational behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, compact sentence that front-loads the core action. It contains no filler words. However, it is slightly too terse, missing structured elements like parameter hints or usage context. It earns points for efficiency but loses one for not having any additional structure like bullet points or examples.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that there is no output schema, no annotations, and three parameters with zero schema descriptions, the description should compensate by explaining the return value (scraped content format), error cases, and prerequisites. It provides none of these. The tool's complexity (navigating and scraping) demands a richer description than what is given.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must add meaning for all three parameters. It does partially: it connects 'selector' and 'link_text' to 'CSS selector or link text'. However, the 'url' parameter is left undefined—it could be the page containing the link or the target URL—and no format or constraints are given. The description provides only marginal value beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (click) and the resource (specified link via CSS selector or link text), and it adds the additional step of scraping the target page content. This differentiates it from sibling tools like scrape_page (which just scrapes without clicking) and crawl_site (which follows multiple links). The purpose is specific and unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this tool versus alternatives like scrape_page or crawl_site. The description only states what it does, with no context about prerequisites (e.g., needing to be on a page first), nor exclusions (e.g., not for links that require authentication). The agent must infer usage from the name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
BFS 全站爬取（谷歌爬虫式）：从 start_url 遍历站内所有内链。
筛选：published_after/before（ISO 时间，发布时间优先、抓取时间兜底）、 title_contains/title_exclude（标题包含）、url_pattern（glob/regex 模式）。 默认遵守 robots.txt 并按 0.5s/请求限速，可关闭。 结果 JSON 返回并持久化到 SQLite。
| Name | Required | Description | Default |
|---|---|---|---|
| max_depth | No | ||
| max_pages | No | ||
| start_url | Yes | ||
| rate_limit | No | ||
| url_pattern | No | ||
| title_exclude | No | ||
| respect_robots | No | ||
| title_contains | No | ||
| published_after | No | ||
| include_external | No | ||
| published_before | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden. It discloses BFS traversal, default robots.txt respect, rate limiting (0.5s), JSON output persistence to SQLite, and fallback behavior for date filters. However, it does not mention error handling or authentication requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is efficient, using a few lines to convey purpose and filters. It is front-loaded with the main action and uses bullet-style listing for filters. No redundant text, though structural improvements (e.g., separating behavior from parameters) could enhance readability.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 11 parameters and no output schema, the description covers core crawl behavior and key filters but lacks details on crawl limits (max_depth, max_pages), output structure beyond 'JSON', and how to access persisted data (likely via query_crawls). The overall completeness is adequate but not thorough.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so description must compensate. It explains date filters (ISO time, fallback), title filters, url_pattern (glob/regex), and toggles for robots.txt and rate limit. However, it omits max_depth, max_pages, and include_external. While start_url is obvious, the missing parameters reduce completeness.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that the tool performs a BFS full-site crawl starting from a URL, traversing all internal links. It distinguishes itself from sibling tools like scrape_page (single page) and query_crawls (querying stored results) by specifying the crawling algorithm and scope.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for full-site crawling but does not explicitly contrast with siblings or provide when-not-to-use scenarios. While filters and defaults are listed, there is no direct guidance on selecting this tool over scrape_page or download_file for different tasks.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
下载页面文件（图片/文档等）到 E:\Reasonix-项目\crawler-output。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| filename | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations are absent, so description carries full burden. Only mentions destination path, but omits critical behaviors: overwrite policy, file type restrictions, error handling, or confirmation that it downloads from a given URL.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence, front-loaded with verb, no fluff. However, it is underspecified, which slightly reduces effectiveness despite brevity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 2-parameter tool with no output schema, description fails to explain how to invoke correctly (e.g., do both parameters need to be specified? What is the output or success indication?). Lacks critical context for reliable agent use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has 0% description coverage for parameters; description adds no explanation for 'url' (expected format/sources) or 'filename' (override behavior). Agent has no semantic help beyond parameter names and types.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states verb 'download' and resource 'page files' with specific destination path. This distinguishes it from sibling tools like scrape_page or click_link, which don't involve file saving.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool vs alternatives (e.g., when to download vs scrape vs summarize). Lacks context about prerequisites or suitable scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
查询已持久化的爬取结果（SQLite），供其它项目复用。
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| success_only | No | ||
| url_contains | No | ||
| title_contains | No | ||
| published_after | No | ||
| published_before | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It implicitly suggests a read operation ('query') but does not explicitly state that it is non-destructive, safe to call repeatedly, or what happens with empty results. No side effects, auth needs, or rate limits are disclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence in Chinese, which is concise but not well-structured. It front-loads the verb but lacks any structure or additional sentences to elaborate on usage or parameters. Every sentence should earn its place, and here there is only one sentence that could be more informative.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (6 optional parameters, no output schema, no annotations), the description is severely incomplete. It does not explain return format, how filters combine, or what 'success_only' means. The agent would struggle to use this tool effectively based solely on the description.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, yet the description adds no explanation for any of the 6 parameters (limit, success_only, url_contains, etc.). The description offers zero value beyond the parameter names, leaving the agent to infer their meaning without any context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'query persisted crawl results (SQLite) for reuse by other projects,' identifying the verb (query) and resource (crawl results). It distinguishes from sibling tools like crawl_site (creation) and scrape_page (web scraping). However, it could be more specific about the scope of results and the read-only nature.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives (e.g., crawl_site for creating crawls, scrape_page for live scraping). There are no prerequisites, exclusions, or context for when this query is appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
抓取单个页面，返回 markdown、标题、发布时间与链接列表，并持久化。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| respect_robots | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses persistence ('并持久化') and the return types, but it does not mention mutability, side effects, or operational details like rate limits or authentication requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence that front-loads the action and outputs. Every word earns its place; no redundancy or fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the absence of an output schema and annotations, the description is adequate but incomplete. It lists return types and persistence but omits details on error handling, parameter behavior, and the exact structure of the returned data.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% with no parameter descriptions. The description only implies the 'url' parameter through the tool's purpose but does not explain the 'respect_robots' parameter at all. This gap leaves the agent without guidance on an important scraping behavior.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states 'scrape a single page' and lists the returned content (markdown, title, publish time, links). It clearly distinguishes from sibling tools like crawl_site (multiple pages) and summarize_page (summarization).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies single-page usage via '单个页面', but it does not explicitly state when to use this tool versus alternatives like crawl_site or click_link. No exclusions or prerequisites are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
概括页面内容。mode: auto（LLM 优先，失败回落本地）/ llm / local。
llm_provider 如 "deepseek/deepseek-chat" 或 "openai/gpt-4o-mini"（LiteLLM 格式）； llm_api_key_env 指定 API key 的环境变量名（默认 DEEPSEEK_API_KEY）。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| mode | No | auto | |
| llm_provider | No | ||
| llm_api_key_env | No | DEEPSEEK_API_KEY |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It discloses key behavioral traits: the auto mode tries LLM first and falls back to local on failure, and it specifies the llm_provider format and llm_api_key_env default. This adds useful context beyond the schema, though it stops short of explaining error scenarios or output formats.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and well-structured: it opens with the main purpose, then details modes and provider parameters in a clear, scannable format. Every sentence adds value with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 4 parameters, no output schema, and no annotations, the description covers the essential functional aspects: purpose, mode behaviors, and provider configuration. It doesn't mention return value or edge cases, but this is not critical for basic invocation and is adequate given the tool's simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage, so the description's explanations are essential. It defines allowed mode values, provides concrete LiteLLM format examples for llm_provider, and states the default for llm_api_key_env. This compensates well for the schema's lack of descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states '概括页面内容' (summarize page content), identifying a specific verb (summarize) and resource (page). This purpose is distinct from sibling tools like scrape_page or crawl_site, making it easy for an agent to know what this tool does.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not explicitly state when to use this tool instead of alternatives like scrape_page or crawl_site. It does provide mode-specific guidance (auto/llm/local) and parameter details, but lacks explicit when-to-use or when-not-to-use statements relative to sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0click_linkcrawl_sitedownload_filequery_crawlsscrape_pagesummarize_pageScored across 6 tools
Each tool has a clearly distinct purpose: full site crawling, single page scraping, summarization, link clicking, file downloading, and querying. No overlap or ambiguity.
All tools follow a consistent verb_noun pattern in snake_case (e.g., crawl_site, scrape_page, query_crawls), making the tool set predictable and easy to navigate.
With 6 tools, the server is well-scoped. It covers the essential operations for a site crawler without unnecessary bloat or missing functionality.
The tool set covers core workflows: crawling, scraping, summarizing, interacting, downloading, and querying. Minor gaps like explicit crawl session management or update/delete operations exist, but the surface is largely complete for typical use cases.
Web search, URL content extraction to Markdown, site mapping, and recursive web crawler.
Scrape any URL into clean LLM-ready Markdown, or crawl a whole site.
Crawl a site for broken links, 404s, dead images, redirect chains and slow pages, with sources
Crawl a site for broken links, redirects and missing SEO tags. Pay per page on Apify.
