Un servidor MCP de rastreo completo del sitio construido sobre crawl4ai (Apache-2.0). Rastrea todos los enlaces internos de un sitio web como un spider de motor de búsqueda (BFS), filtra páginas por fecha de publicación / título / URL, y puede operar sobre páginas rastreadas (resumir, hacer clic en enlaces, descargar archivos). Los resultados se devuelven como JSON y se persisten en SQLite para su reutilización por cualquier proyecto.
Resumen en español: un MCP de rastreo completo de enlaces internos «como el crawler de Google». Basado en crawl4ai con encapsulación propia, admite recorrido BFS completo del sitio, filtrado por fecha/título/URL, operaciones en páginas (resumen LLM/clic en enlaces/descarga de archivos), resultados en JSON + persistencia en SQLite. Regístrelo en Reasonix / Claude Desktop / Cursor o cualquier cliente MCP para reutilizarlo globalmente.
✅ Rastreo BFS completo del sitio — recorre todos los enlaces internos (max_depth / max_page controlables)
✅ Filtro de tiempo — extrae la fecha de publicación de metadatos de página / JSON-LD / URL primero,
recurre a la hora de rastreo cuando no está disponible (resultado etiquetado con time_source)
✅ Filtro de título — incluir/excluir por palabra clave en el título (sin distinción de mayúsculas/minúsculas)
✅ Filtros de patrón URL y dominio — coincidencia glob/regex de URL, restricción al mismo dominio
✅ Rastreo educado — respeta robots.txt y limita la tasa por defecto (configurable)
✅ Operaciones en páginas — resumen LLM (LiteLLM: DeepSeek / GLM / OpenAI…, fallback local), hacer clic en un enlace específico (selector CSS o texto del enlace), descargar archivos
✅ Persistencia en SQLite — herramienta query_crawls para reutilizar datos rastreados en proyectos posteriores
Related MCP server: Spider MCP Server
Herramienta | Descripción |
| Rastreo BFS completo del sitio. Parámetros: |
| Extraer una sola página (markdown / título / fecha de publicación / enlaces) |
| Resumir el contenido de la página. |
| Hacer clic en un enlace dentro de la página ( |
| Descargar archivos de la página al directorio de salida (por defecto |
| Consultar resultados de rastreo persistidos desde SQLite (filtros de título/URL/tiempo) |
Python ≥ 3.12 (probado en 3.12.13)
uv recomendado (opcional — también funciona con pip normal)
Navegadores Playwright: python -m playwright install chromium
(o apunte PLAYWRIGHT_BROWSERS_PATH a una instalación de navegador existente)
# 1. Create environment & install
uv venv .venv --python 3.12
uv pip install --python .venv\Scripts\python.exe crawl4ai "mcp>=1.2,<2"
uv pip install --python .venv\Scripts\python.exe -e .
# 2. Install browser (once)
.venv\Scripts\python.exe -m playwright install chromium
# 3. Register as MCP server (example for Reasonix config.toml)
[[plugins]]
name = "site-crawler-mcp"
type = "stdio"
command = "C:\\path\\to\\site-crawler-mcp\\.venv\\Scripts\\python.exe"
args = ["-m", "site_crawler_mcp.server"]Para Claude Desktop / Cursor, agregue los mismos command/args bajo mcpServers en sus archivos de configuración.
import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
async def main():
params = StdioServerParameters(command="python", args=["-m", "site_crawler_mcp.server"])
async with stdio_client(params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
res = await session.call_tool("crawl_site", {
"start_url": "https://example.com",
"max_depth": 2,
"max_pages": 20,
"title_contains": "Example",
})
print(res.content[0].text)
asyncio.run(main())Los filtros de URL de crawl4ai solo funcionan sobre las URL, por lo que aquí se implementa el filtrado a nivel de contenido:
Poda a nivel de URL — TimeRangeFilter / TitleFilter (patrones de fecha en URL, palabras clave en URL)
A nivel de contenido — después de obtener cada página, la fecha de publicación se extrae de
<meta property="article:published_time">, JSON-LD datePublished, <time datetime>,
o YYYY/MM/DD en la URL. Si no se encuentra ninguna, se usa la hora de rastreo (decisión documentada como time_source).
Respeta robots.txt y límites de tasa por defecto para evitar estresar los sitios de destino o ser bloqueado por IP.
Para aprendizaje / investigación / sus propios sitios; siga los Términos de Servicio de los sitios de destino y las leyes locales.
Construido sobre crawl4ai (Apache-2.0).
点击页面内指定链接（CSS selector 或链接文本）并抓取目标页内容。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| selector | No | ||
| link_text | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden for behavioral disclosure. It mentions 'click and scrape' but omits critical traits: what happens if the link is not found, whether JavaScript-triggered navigation is supported, whether it waits for page load, or if it modifies browser state. The description is too minimal to convey the tool's operational behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, compact sentence that front-loads the core action. It contains no filler words. However, it is slightly too terse, missing structured elements like parameter hints or usage context. It earns points for efficiency but loses one for not having any additional structure like bullet points or examples.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that there is no output schema, no annotations, and three parameters with zero schema descriptions, the description should compensate by explaining the return value (scraped content format), error cases, and prerequisites. It provides none of these. The tool's complexity (navigating and scraping) demands a richer description than what is given.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must add meaning for all three parameters. It does partially: it connects 'selector' and 'link_text' to 'CSS selector or link text'. However, the 'url' parameter is left undefined—it could be the page containing the link or the target URL—and no format or constraints are given. The description provides only marginal value beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (click) and the resource (specified link via CSS selector or link text), and it adds the additional step of scraping the target page content. This differentiates it from sibling tools like scrape_page (which just scrapes without clicking) and crawl_site (which follows multiple links). The purpose is specific and unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this tool versus alternatives like scrape_page or crawl_site. The description only states what it does, with no context about prerequisites (e.g., needing to be on a page first), nor exclusions (e.g., not for links that require authentication). The agent must infer usage from the name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
BFS 全站爬取（谷歌爬虫式）：从 start_url 遍历站内所有内链。
筛选：published_after/before（ISO 时间，发布时间优先、抓取时间兜底）、 title_contains/title_exclude（标题包含）、url_pattern（glob/regex 模式）。 默认遵守 robots.txt 并按 0.5s/请求限速，可关闭。 结果 JSON 返回并持久化到 SQLite。
| Name | Required | Description | Default |
|---|---|---|---|
| max_depth | No | ||
| max_pages | No | ||
| start_url | Yes | ||
| rate_limit | No | ||
| url_pattern | No | ||
| title_exclude | No | ||
| respect_robots | No | ||
| title_contains | No | ||
| published_after | No | ||
| include_external | No | ||
| published_before | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden. It discloses BFS traversal, default robots.txt respect, rate limiting (0.5s), JSON output persistence to SQLite, and fallback behavior for date filters. However, it does not mention error handling or authentication requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is efficient, using a few lines to convey purpose and filters. It is front-loaded with the main action and uses bullet-style listing for filters. No redundant text, though structural improvements (e.g., separating behavior from parameters) could enhance readability.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 11 parameters and no output schema, the description covers core crawl behavior and key filters but lacks details on crawl limits (max_depth, max_pages), output structure beyond 'JSON', and how to access persisted data (likely via query_crawls). The overall completeness is adequate but not thorough.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so description must compensate. It explains date filters (ISO time, fallback), title filters, url_pattern (glob/regex), and toggles for robots.txt and rate limit. However, it omits max_depth, max_pages, and include_external. While start_url is obvious, the missing parameters reduce completeness.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that the tool performs a BFS full-site crawl starting from a URL, traversing all internal links. It distinguishes itself from sibling tools like scrape_page (single page) and query_crawls (querying stored results) by specifying the crawling algorithm and scope.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for full-site crawling but does not explicitly contrast with siblings or provide when-not-to-use scenarios. While filters and defaults are listed, there is no direct guidance on selecting this tool over scrape_page or download_file for different tasks.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
下载页面文件（图片/文档等）到 E:\Reasonix-项目\crawler-output。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| filename | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations are absent, so description carries full burden. Only mentions destination path, but omits critical behaviors: overwrite policy, file type restrictions, error handling, or confirmation that it downloads from a given URL.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence, front-loaded with verb, no fluff. However, it is underspecified, which slightly reduces effectiveness despite brevity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 2-parameter tool with no output schema, description fails to explain how to invoke correctly (e.g., do both parameters need to be specified? What is the output or success indication?). Lacks critical context for reliable agent use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has 0% description coverage for parameters; description adds no explanation for 'url' (expected format/sources) or 'filename' (override behavior). Agent has no semantic help beyond parameter names and types.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states verb 'download' and resource 'page files' with specific destination path. This distinguishes it from sibling tools like scrape_page or click_link, which don't involve file saving.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool vs alternatives (e.g., when to download vs scrape vs summarize). Lacks context about prerequisites or suitable scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
查询已持久化的爬取结果（SQLite），供其它项目复用。
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| success_only | No | ||
| url_contains | No | ||
| title_contains | No | ||
| published_after | No | ||
| published_before | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It implicitly suggests a read operation ('query') but does not explicitly state that it is non-destructive, safe to call repeatedly, or what happens with empty results. No side effects, auth needs, or rate limits are disclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence in Chinese, which is concise but not well-structured. It front-loads the verb but lacks any structure or additional sentences to elaborate on usage or parameters. Every sentence should earn its place, and here there is only one sentence that could be more informative.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (6 optional parameters, no output schema, no annotations), the description is severely incomplete. It does not explain return format, how filters combine, or what 'success_only' means. The agent would struggle to use this tool effectively based solely on the description.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, yet the description adds no explanation for any of the 6 parameters (limit, success_only, url_contains, etc.). The description offers zero value beyond the parameter names, leaving the agent to infer their meaning without any context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'query persisted crawl results (SQLite) for reuse by other projects,' identifying the verb (query) and resource (crawl results). It distinguishes from sibling tools like crawl_site (creation) and scrape_page (web scraping). However, it could be more specific about the scope of results and the read-only nature.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives (e.g., crawl_site for creating crawls, scrape_page for live scraping). There are no prerequisites, exclusions, or context for when this query is appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
抓取单个页面，返回 markdown、标题、发布时间与链接列表，并持久化。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| respect_robots | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses persistence ('并持久化') and the return types, but it does not mention mutability, side effects, or operational details like rate limits or authentication requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence that front-loads the action and outputs. Every word earns its place; no redundancy or fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the absence of an output schema and annotations, the description is adequate but incomplete. It lists return types and persistence but omits details on error handling, parameter behavior, and the exact structure of the returned data.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% with no parameter descriptions. The description only implies the 'url' parameter through the tool's purpose but does not explain the 'respect_robots' parameter at all. This gap leaves the agent without guidance on an important scraping behavior.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states 'scrape a single page' and lists the returned content (markdown, title, publish time, links). It clearly distinguishes from sibling tools like crawl_site (multiple pages) and summarize_page (summarization).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies single-page usage via '单个页面', but it does not explicitly state when to use this tool versus alternatives like crawl_site or click_link. No exclusions or prerequisites are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
概括页面内容。mode: auto（LLM 优先，失败回落本地）/ llm / local。
llm_provider 如 "deepseek/deepseek-chat" 或 "openai/gpt-4o-mini"（LiteLLM 格式）； llm_api_key_env 指定 API key 的环境变量名（默认 DEEPSEEK_API_KEY）。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| mode | No | auto | |
| llm_provider | No | ||
| llm_api_key_env | No | DEEPSEEK_API_KEY |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It discloses key behavioral traits: the auto mode tries LLM first and falls back to local on failure, and it specifies the llm_provider format and llm_api_key_env default. This adds useful context beyond the schema, though it stops short of explaining error scenarios or output formats.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and well-structured: it opens with the main purpose, then details modes and provider parameters in a clear, scannable format. Every sentence adds value with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 4 parameters, no output schema, and no annotations, the description covers the essential functional aspects: purpose, mode behaviors, and provider configuration. It doesn't mention return value or edge cases, but this is not critical for basic invocation and is adequate given the tool's simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage, so the description's explanations are essential. It defines allowed mode values, provides concrete LiteLLM format examples for llm_provider, and states the default for llm_api_key_env. This compensates well for the schema's lack of descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states '概括页面内容' (summarize page content), identifying a specific verb (summarize) and resource (page). This purpose is distinct from sibling tools like scrape_page or crawl_site, making it easy for an agent to know what this tool does.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not explicitly state when to use this tool instead of alternatives like scrape_page or crawl_site. It does provide mode-specific guidance (auto/llm/local) and parameter details, but lacks explicit when-to-use or when-not-to-use statements relative to sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
v0.1.0click_linkcrawl_sitedownload_filequery_crawlsscrape_pagesummarize_pageScored across 6 tools
Each tool has a clearly distinct purpose: full site crawling, single page scraping, summarization, link clicking, file downloading, and querying. No overlap or ambiguity.
All tools follow a consistent verb_noun pattern in snake_case (e.g., crawl_site, scrape_page, query_crawls), making the tool set predictable and easy to navigate.
With 6 tools, the server is well-scoped. It covers the essential operations for a site crawler without unnecessary bloat or missing functionality.
The tool set covers core workflows: crawling, scraping, summarizing, interacting, downloading, and querying. Minor gaps like explicit crawl session management or update/delete operations exist, but the surface is largely complete for typical use cases.
Web search, URL content extraction to Markdown, site mapping, and recursive web crawler.
Scrape any URL into clean LLM-ready Markdown, or crawl a whole site.
Crawl a site for broken links, 404s, dead images, redirect chains and slow pages, with sources
Crawl a site for broken links, redirects and missing SEO tags. Pay per page on Apify.
