Сайтовый краулер MCP сервер, построенный на crawl4ai (Apache-2.0). Он обходит все внутренние ссылки сайта как паук поисковой системы (BFS), фильтрует страницы по времени публикации / заголовку / URL и может выполнять операции над собранными страницами (суммировать, нажимать ссылки, скачивать файлы). Результаты возвращаются в формате JSON и сохраняются в SQLite для повторного использования любым проектом.
中文简介：一个「像谷歌爬虫一样」的全站内链爬虫 MCP。基于 crawl4ai 自研封装， 支持 BFS 全站遍历、时间/标题/URL 筛选、页面操作（LLM 概括/点击链接/下载文件）、 结果 JSON 返回 + SQLite 持久化。注册到 Reasonix / Claude Desktop / Cursor 等任意 MCP 客户端即可全局复用。
✅ Обход всего сайта BFS — обходит все внутренние ссылки (max_depth / max_pages контролируемы)
✅ Фильтр по времени — извлекает время публикации из метаданных страницы / JSON-LD / URL в первую очередь,
при отсутствии использует время обхода (результат помечен time_source)
✅ Фильтр по заголовку — включение / исключение по ключевому слову в заголовке (без учета регистра)
✅ Фильтры по шаблону URL и домену — сопоставление с glob/regex URL, ограничение по тому же домену
✅ Вежливый обход — уважает robots.txt и ограничивает скорость по умолчанию (можно отключить)
✅ Операции над страницами — LLM суммаризация (LiteLLM: DeepSeek / GLM / OpenAI…, локальный запасной вариант), нажатие на конкретную ссылку (CSS-селектор или текст ссылки), скачивание файлов
✅ Сохранение в SQLite — инструмент query_crawls для повторного использования собранных данных в последующих проектах
Related MCP server: Spider MCP Server
Инструмент | Описание |
| BFS обход всего сайта. Параметры: |
| Сбор одной страницы (markdown / заголовок / время публикации / ссылки) |
| Суммаризация содержимого страницы. |
| Нажатие на ссылку внутри страницы ( |
| Скачивание файлов со страницы в выходную директорию (по умолчанию |
| Запрос сохранённых результатов обхода из SQLite (фильтры по заголовку/URL/времени) |
Python ≥ 3.12 (протестировано на 3.12.13)
Рекомендуется uv (также работает обычный pip)
Браузеры Playwright: python -m playwright install chromium
(или укажите PLAYWRIGHT_BROWSERS_PATH на существующий установленный браузер)
# 1. Create environment & install
uv venv .venv --python 3.12
uv pip install --python .venv\Scripts\python.exe crawl4ai "mcp>=1.2,<2"
uv pip install --python .venv\Scripts\python.exe -e .
# 2. Install browser (once)
.venv\Scripts\python.exe -m playwright install chromium
# 3. Register as MCP server (example for Reasonix config.toml)
[[plugins]]
name = "site-crawler-mcp"
type = "stdio"
command = "C:\\path\\to\\site-crawler-mcp\\.venv\\Scripts\\python.exe"
args = ["-m", "site_crawler_mcp.server"]Для Claude Desktop / Cursor добавьте те же command/args в раздел mcpServers в их конфигурационных файлах.
import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
async def main():
params = StdioServerParameters(command="python", args=["-m", "site_crawler_mcp.server"])
async with stdio_client(params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
res = await session.call_tool("crawl_site", {
"start_url": "https://example.com",
"max_depth": 2,
"max_pages": 20,
"title_contains": "Example",
})
print(res.content[0].text)
asyncio.run(main())Фильтры URL в crawl4ai работают только по URL, поэтому фильтрация на уровне содержимого реализована здесь:
Обрезка на уровне URL — TimeRangeFilter / TitleFilter (шаблоны даты в URL, ключевые слова в URL)
На уровне содержимого — после получения каждой страницы время публикации извлекается из
<meta property="article:published_time">, JSON-LD datePublished, <time datetime>,
или YYYY/MM/DD в URL. Если ничего не найдено, используется время обхода (решение документируется как time_source).
По умолчанию соблюдает robots.txt и ограничения скорости, чтобы не нагружать целевые сайты и не попасть в IP-бан.
Для обучения / исследований / ваших собственных сайтов; пожалуйста, соблюдайте условия использования целевых сайтов и местные законы.
Основано на crawl4ai (Apache-2.0).
点击页面内指定链接（CSS selector 或链接文本）并抓取目标页内容。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| selector | No | ||
| link_text | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden for behavioral disclosure. It mentions 'click and scrape' but omits critical traits: what happens if the link is not found, whether JavaScript-triggered navigation is supported, whether it waits for page load, or if it modifies browser state. The description is too minimal to convey the tool's operational behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, compact sentence that front-loads the core action. It contains no filler words. However, it is slightly too terse, missing structured elements like parameter hints or usage context. It earns points for efficiency but loses one for not having any additional structure like bullet points or examples.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that there is no output schema, no annotations, and three parameters with zero schema descriptions, the description should compensate by explaining the return value (scraped content format), error cases, and prerequisites. It provides none of these. The tool's complexity (navigating and scraping) demands a richer description than what is given.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must add meaning for all three parameters. It does partially: it connects 'selector' and 'link_text' to 'CSS selector or link text'. However, the 'url' parameter is left undefined—it could be the page containing the link or the target URL—and no format or constraints are given. The description provides only marginal value beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (click) and the resource (specified link via CSS selector or link text), and it adds the additional step of scraping the target page content. This differentiates it from sibling tools like scrape_page (which just scrapes without clicking) and crawl_site (which follows multiple links). The purpose is specific and unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this tool versus alternatives like scrape_page or crawl_site. The description only states what it does, with no context about prerequisites (e.g., needing to be on a page first), nor exclusions (e.g., not for links that require authentication). The agent must infer usage from the name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
BFS 全站爬取（谷歌爬虫式）：从 start_url 遍历站内所有内链。
筛选：published_after/before（ISO 时间，发布时间优先、抓取时间兜底）、 title_contains/title_exclude（标题包含）、url_pattern（glob/regex 模式）。 默认遵守 robots.txt 并按 0.5s/请求限速，可关闭。 结果 JSON 返回并持久化到 SQLite。
| Name | Required | Description | Default |
|---|---|---|---|
| max_depth | No | ||
| max_pages | No | ||
| start_url | Yes | ||
| rate_limit | No | ||
| url_pattern | No | ||
| title_exclude | No | ||
| respect_robots | No | ||
| title_contains | No | ||
| published_after | No | ||
| include_external | No | ||
| published_before | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden. It discloses BFS traversal, default robots.txt respect, rate limiting (0.5s), JSON output persistence to SQLite, and fallback behavior for date filters. However, it does not mention error handling or authentication requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is efficient, using a few lines to convey purpose and filters. It is front-loaded with the main action and uses bullet-style listing for filters. No redundant text, though structural improvements (e.g., separating behavior from parameters) could enhance readability.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 11 parameters and no output schema, the description covers core crawl behavior and key filters but lacks details on crawl limits (max_depth, max_pages), output structure beyond 'JSON', and how to access persisted data (likely via query_crawls). The overall completeness is adequate but not thorough.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so description must compensate. It explains date filters (ISO time, fallback), title filters, url_pattern (glob/regex), and toggles for robots.txt and rate limit. However, it omits max_depth, max_pages, and include_external. While start_url is obvious, the missing parameters reduce completeness.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that the tool performs a BFS full-site crawl starting from a URL, traversing all internal links. It distinguishes itself from sibling tools like scrape_page (single page) and query_crawls (querying stored results) by specifying the crawling algorithm and scope.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for full-site crawling but does not explicitly contrast with siblings or provide when-not-to-use scenarios. While filters and defaults are listed, there is no direct guidance on selecting this tool over scrape_page or download_file for different tasks.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
下载页面文件（图片/文档等）到 E:\Reasonix-项目\crawler-output。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| filename | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations are absent, so description carries full burden. Only mentions destination path, but omits critical behaviors: overwrite policy, file type restrictions, error handling, or confirmation that it downloads from a given URL.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence, front-loaded with verb, no fluff. However, it is underspecified, which slightly reduces effectiveness despite brevity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 2-parameter tool with no output schema, description fails to explain how to invoke correctly (e.g., do both parameters need to be specified? What is the output or success indication?). Lacks critical context for reliable agent use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has 0% description coverage for parameters; description adds no explanation for 'url' (expected format/sources) or 'filename' (override behavior). Agent has no semantic help beyond parameter names and types.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states verb 'download' and resource 'page files' with specific destination path. This distinguishes it from sibling tools like scrape_page or click_link, which don't involve file saving.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool vs alternatives (e.g., when to download vs scrape vs summarize). Lacks context about prerequisites or suitable scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
查询已持久化的爬取结果（SQLite），供其它项目复用。
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| success_only | No | ||
| url_contains | No | ||
| title_contains | No | ||
| published_after | No | ||
| published_before | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It implicitly suggests a read operation ('query') but does not explicitly state that it is non-destructive, safe to call repeatedly, or what happens with empty results. No side effects, auth needs, or rate limits are disclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence in Chinese, which is concise but not well-structured. It front-loads the verb but lacks any structure or additional sentences to elaborate on usage or parameters. Every sentence should earn its place, and here there is only one sentence that could be more informative.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (6 optional parameters, no output schema, no annotations), the description is severely incomplete. It does not explain return format, how filters combine, or what 'success_only' means. The agent would struggle to use this tool effectively based solely on the description.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, yet the description adds no explanation for any of the 6 parameters (limit, success_only, url_contains, etc.). The description offers zero value beyond the parameter names, leaving the agent to infer their meaning without any context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'query persisted crawl results (SQLite) for reuse by other projects,' identifying the verb (query) and resource (crawl results). It distinguishes from sibling tools like crawl_site (creation) and scrape_page (web scraping). However, it could be more specific about the scope of results and the read-only nature.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives (e.g., crawl_site for creating crawls, scrape_page for live scraping). There are no prerequisites, exclusions, or context for when this query is appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
抓取单个页面，返回 markdown、标题、发布时间与链接列表，并持久化。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| respect_robots | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses persistence ('并持久化') and the return types, but it does not mention mutability, side effects, or operational details like rate limits or authentication requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence that front-loads the action and outputs. Every word earns its place; no redundancy or fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the absence of an output schema and annotations, the description is adequate but incomplete. It lists return types and persistence but omits details on error handling, parameter behavior, and the exact structure of the returned data.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% with no parameter descriptions. The description only implies the 'url' parameter through the tool's purpose but does not explain the 'respect_robots' parameter at all. This gap leaves the agent without guidance on an important scraping behavior.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states 'scrape a single page' and lists the returned content (markdown, title, publish time, links). It clearly distinguishes from sibling tools like crawl_site (multiple pages) and summarize_page (summarization).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies single-page usage via '单个页面', but it does not explicitly state when to use this tool versus alternatives like crawl_site or click_link. No exclusions or prerequisites are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
概括页面内容。mode: auto（LLM 优先，失败回落本地）/ llm / local。
llm_provider 如 "deepseek/deepseek-chat" 或 "openai/gpt-4o-mini"（LiteLLM 格式）； llm_api_key_env 指定 API key 的环境变量名（默认 DEEPSEEK_API_KEY）。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| mode | No | auto | |
| llm_provider | No | ||
| llm_api_key_env | No | DEEPSEEK_API_KEY |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It discloses key behavioral traits: the auto mode tries LLM first and falls back to local on failure, and it specifies the llm_provider format and llm_api_key_env default. This adds useful context beyond the schema, though it stops short of explaining error scenarios or output formats.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and well-structured: it opens with the main purpose, then details modes and provider parameters in a clear, scannable format. Every sentence adds value with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 4 parameters, no output schema, and no annotations, the description covers the essential functional aspects: purpose, mode behaviors, and provider configuration. It doesn't mention return value or edge cases, but this is not critical for basic invocation and is adequate given the tool's simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage, so the description's explanations are essential. It defines allowed mode values, provides concrete LiteLLM format examples for llm_provider, and states the default for llm_api_key_env. This compensates well for the schema's lack of descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states '概括页面内容' (summarize page content), identifying a specific verb (summarize) and resource (page). This purpose is distinct from sibling tools like scrape_page or crawl_site, making it easy for an agent to know what this tool does.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not explicitly state when to use this tool instead of alternatives like scrape_page or crawl_site. It does provide mode-specific guidance (auto/llm/local) and parameter details, but lacks explicit when-to-use or when-not-to-use statements relative to sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0click_linkcrawl_sitedownload_filequery_crawlsscrape_pagesummarize_pageScored across 6 tools
Each tool has a clearly distinct purpose: full site crawling, single page scraping, summarization, link clicking, file downloading, and querying. No overlap or ambiguity.
All tools follow a consistent verb_noun pattern in snake_case (e.g., crawl_site, scrape_page, query_crawls), making the tool set predictable and easy to navigate.
With 6 tools, the server is well-scoped. It covers the essential operations for a site crawler without unnecessary bloat or missing functionality.
The tool set covers core workflows: crawling, scraping, summarizing, interacting, downloading, and querying. Minor gaps like explicit crawl session management or update/delete operations exist, but the surface is largely complete for typical use cases.
Web search, URL content extraction to Markdown, site mapping, and recursive web crawler.
Scrape any URL into clean LLM-ready Markdown, or crawl a whole site.
Crawl a site for broken links, 404s, dead images, redirect chains and slow pages, with sources
Crawl a site for broken links, redirects and missing SEO tags. Pay per page on Apify.
