Persists crawled site data to a SQLite database for reuse across projects, with tools to query and filter previously crawled results.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@site-crawler-mcp crawl https://example.com and show me all pages about products"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
A site-wide crawler MCP server built on crawl4ai (Apache-2.0). It crawls every internal link of a website like a search-engine spider (BFS), filters pages by publish time / title / URL, and can operate on crawled pages (summarize, click links, download files). Results are returned as JSON and persisted to SQLite for reuse by any project.
中文简介：一个「像谷歌爬虫一样」的全站内链爬虫 MCP。基于 crawl4ai 自研封装， 支持 BFS 全站遍历、时间/标题/URL 筛选、页面操作（LLM 概括/点击链接/下载文件）、 结果 JSON 返回 + SQLite 持久化。注册到 Reasonix / Claude Desktop / Cursor 等任意 MCP 客户端即可全局复用。
✅ Site-wide BFS crawl — traverse all internal links (max_depth / max_pages controllable)
✅ Time filter — extract publish time from page meta / JSON-LD / URL first,
fall back to crawl time when unavailable (result tagged with time_source)
✅ Title filter — include / exclude by title keyword (case-insensitive)
✅ URL pattern & domain filters — glob/regex URL matching, same-domain restriction
✅ Polite crawling — respects robots.txt and rate-limits by default (toggleable)
✅ Page operations — LLM summarize (LiteLLM: DeepSeek / GLM / OpenAI…, local fallback), click a specific link (CSS selector or link text), download files
✅ SQLite persistence — query_crawls tool to reuse crawled data in later projects
Related MCP server: Spider MCP Server
Tool | Description |
| BFS site-wide crawl. Params: |
| Scrape a single page (markdown / title / publish time / links) |
| Summarize page content. |
| Click a link inside the page ( |
| Download page files to output dir (default |
| Query persisted crawl results from SQLite (title/URL/time filters) |
Python ≥ 3.12 (tested on 3.12.13)
uv recommended (optional — plain pip works too)
Playwright browsers: python -m playwright install chromium
(or point PLAYWRIGHT_BROWSERS_PATH at an existing browser install)
# 1. Create environment & install
uv venv .venv --python 3.12
uv pip install --python .venv\Scripts\python.exe crawl4ai "mcp>=1.2,<2"
uv pip install --python .venv\Scripts\python.exe -e .
# 2. Install browser (once)
.venv\Scripts\python.exe -m playwright install chromium
# 3. Register as MCP server (example for Reasonix config.toml)
[[plugins]]
name = "site-crawler-mcp"
type = "stdio"
command = "C:\\path\\to\\site-crawler-mcp\\.venv\\Scripts\\python.exe"
args = ["-m", "site_crawler_mcp.server"]For Claude Desktop / Cursor, add the same command/args under mcpServers in their config files.
import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
async def main():
params = StdioServerParameters(command="python", args=["-m", "site_crawler_mcp.server"])
async with stdio_client(params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
res = await session.call_tool("crawl_site", {
"start_url": "https://example.com",
"max_depth": 2,
"max_pages": 20,
"title_contains": "Example",
})
print(res.content[0].text)
asyncio.run(main())crawl4ai's URL filters work on URLs only, so content-level filtering is implemented here:
URL-level pruning — TimeRangeFilter / TitleFilter (URL date patterns, URL keywords)
Content-level — after each page is fetched, publish time is extracted from
<meta property="article:published_time">, JSON-LD datePublished, <time datetime>,
or YYYY/MM/DD in the URL. If none found, crawl time is used (decision documented as time_source).
Respects robots.txt and rate limits by default to avoid stressing target sites or getting IP-banned.
For learning / research / your own sites; please follow target sites' ToS and local laws.
Built on crawl4ai (Apache-2.0).
点击页面内指定链接（CSS selector 或链接文本）并抓取目标页内容。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| selector | No | ||
| link_text | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden for behavioral disclosure. It mentions 'click and scrape' but omits critical traits: what happens if the link is not found, whether JavaScript-triggered navigation is supported, whether it waits for page load, or if it modifies browser state. The description is too minimal to convey the tool's operational behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, compact sentence that front-loads the core action. It contains no filler words. However, it is slightly too terse, missing structured elements like parameter hints or usage context. It earns points for efficiency but loses one for not having any additional structure like bullet points or examples.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that there is no output schema, no annotations, and three parameters with zero schema descriptions, the description should compensate by explaining the return value (scraped content format), error cases, and prerequisites. It provides none of these. The tool's complexity (navigating and scraping) demands a richer description than what is given.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must add meaning for all three parameters. It does partially: it connects 'selector' and 'link_text' to 'CSS selector or link text'. However, the 'url' parameter is left undefined—it could be the page containing the link or the target URL—and no format or constraints are given. The description provides only marginal value beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (click) and the resource (specified link via CSS selector or link text), and it adds the additional step of scraping the target page content. This differentiates it from sibling tools like scrape_page (which just scrapes without clicking) and crawl_site (which follows multiple links). The purpose is specific and unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to use this tool versus alternatives like scrape_page or crawl_site. The description only states what it does, with no context about prerequisites (e.g., needing to be on a page first), nor exclusions (e.g., not for links that require authentication). The agent must infer usage from the name alone.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
BFS 全站爬取（谷歌爬虫式）：从 start_url 遍历站内所有内链。
筛选：published_after/before（ISO 时间，发布时间优先、抓取时间兜底）、 title_contains/title_exclude（标题包含）、url_pattern（glob/regex 模式）。 默认遵守 robots.txt 并按 0.5s/请求限速，可关闭。 结果 JSON 返回并持久化到 SQLite。
| Name | Required | Description | Default |
|---|---|---|---|
| max_depth | No | ||
| max_pages | No | ||
| start_url | Yes | ||
| rate_limit | No | ||
| url_pattern | No | ||
| title_exclude | No | ||
| respect_robots | No | ||
| title_contains | No | ||
| published_after | No | ||
| include_external | No | ||
| published_before | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries full burden. It discloses BFS traversal, default robots.txt respect, rate limiting (0.5s), JSON output persistence to SQLite, and fallback behavior for date filters. However, it does not mention error handling or authentication requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is efficient, using a few lines to convey purpose and filters. It is front-loaded with the main action and uses bullet-style listing for filters. No redundant text, though structural improvements (e.g., separating behavior from parameters) could enhance readability.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given 11 parameters and no output schema, the description covers core crawl behavior and key filters but lacks details on crawl limits (max_depth, max_pages), output structure beyond 'JSON', and how to access persisted data (likely via query_crawls). The overall completeness is adequate but not thorough.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so description must compensate. It explains date filters (ISO time, fallback), title filters, url_pattern (glob/regex), and toggles for robots.txt and rate limit. However, it omits max_depth, max_pages, and include_external. While start_url is obvious, the missing parameters reduce completeness.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that the tool performs a BFS full-site crawl starting from a URL, traversing all internal links. It distinguishes itself from sibling tools like scrape_page (single page) and query_crawls (querying stored results) by specifying the crawling algorithm and scope.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for full-site crawling but does not explicitly contrast with siblings or provide when-not-to-use scenarios. While filters and defaults are listed, there is no direct guidance on selecting this tool over scrape_page or download_file for different tasks.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
下载页面文件（图片/文档等）到 E:\Reasonix-项目\crawler-output。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| filename | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations are absent, so description carries full burden. Only mentions destination path, but omits critical behaviors: overwrite policy, file type restrictions, error handling, or confirmation that it downloads from a given URL.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Single sentence, front-loaded with verb, no fluff. However, it is underspecified, which slightly reduces effectiveness despite brevity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 2-parameter tool with no output schema, description fails to explain how to invoke correctly (e.g., do both parameters need to be specified? What is the output or success indication?). Lacks critical context for reliable agent use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has 0% description coverage for parameters; description adds no explanation for 'url' (expected format/sources) or 'filename' (override behavior). Agent has no semantic help beyond parameter names and types.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states verb 'download' and resource 'page files' with specific destination path. This distinguishes it from sibling tools like scrape_page or click_link, which don't involve file saving.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool vs alternatives (e.g., when to download vs scrape vs summarize). Lacks context about prerequisites or suitable scenarios.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
查询已持久化的爬取结果（SQLite），供其它项目复用。
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| success_only | No | ||
| url_contains | No | ||
| title_contains | No | ||
| published_after | No | ||
| published_before | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It implicitly suggests a read operation ('query') but does not explicitly state that it is non-destructive, safe to call repeatedly, or what happens with empty results. No side effects, auth needs, or rate limits are disclosed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single sentence in Chinese, which is concise but not well-structured. It front-loads the verb but lacks any structure or additional sentences to elaborate on usage or parameters. Every sentence should earn its place, and here there is only one sentence that could be more informative.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (6 optional parameters, no output schema, no annotations), the description is severely incomplete. It does not explain return format, how filters combine, or what 'success_only' means. The agent would struggle to use this tool effectively based solely on the description.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, yet the description adds no explanation for any of the 6 parameters (limit, success_only, url_contains, etc.). The description offers zero value beyond the parameter names, leaving the agent to infer their meaning without any context.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'query persisted crawl results (SQLite) for reuse by other projects,' identifying the verb (query) and resource (crawl results). It distinguishes from sibling tools like crawl_site (creation) and scrape_page (web scraping). However, it could be more specific about the scope of results and the read-only nature.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided on when to use this tool versus alternatives (e.g., crawl_site for creating crawls, scrape_page for live scraping). There are no prerequisites, exclusions, or context for when this query is appropriate.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
抓取单个页面，返回 markdown、标题、发布时间与链接列表，并持久化。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| respect_robots | No |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full burden. It discloses persistence ('并持久化') and the return types, but it does not mention mutability, side effects, or operational details like rate limits or authentication requirements.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single concise sentence that front-loads the action and outputs. Every word earns its place; no redundancy or fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the absence of an output schema and annotations, the description is adequate but incomplete. It lists return types and persistence but omits details on error handling, parameter behavior, and the exact structure of the returned data.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% with no parameter descriptions. The description only implies the 'url' parameter through the tool's purpose but does not explain the 'respect_robots' parameter at all. This gap leaves the agent without guidance on an important scraping behavior.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description explicitly states 'scrape a single page' and lists the returned content (markdown, title, publish time, links). It clearly distinguishes from sibling tools like crawl_site (multiple pages) and summarize_page (summarization).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies single-page usage via '单个页面', but it does not explicitly state when to use this tool versus alternatives like crawl_site or click_link. No exclusions or prerequisites are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
概括页面内容。mode: auto（LLM 优先，失败回落本地）/ llm / local。
llm_provider 如 "deepseek/deepseek-chat" 或 "openai/gpt-4o-mini"（LiteLLM 格式）； llm_api_key_env 指定 API key 的环境变量名（默认 DEEPSEEK_API_KEY）。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| mode | No | auto | |
| llm_provider | No | ||
| llm_api_key_env | No | DEEPSEEK_API_KEY |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations provided, the description carries the full burden. It discloses key behavioral traits: the auto mode tries LLM first and falls back to local on failure, and it specifies the llm_provider format and llm_api_key_env default. This adds useful context beyond the schema, though it stops short of explaining error scenarios or output formats.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and well-structured: it opens with the main purpose, then details modes and provider parameters in a clear, scannable format. Every sentence adds value with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 4 parameters, no output schema, and no annotations, the description covers the essential functional aspects: purpose, mode behaviors, and provider configuration. It doesn't mention return value or edge cases, but this is not critical for basic invocation and is adequate given the tool's simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage, so the description's explanations are essential. It defines allowed mode values, provides concrete LiteLLM format examples for llm_provider, and states the default for llm_api_key_env. This compensates well for the schema's lack of descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states '概括页面内容' (summarize page content), identifying a specific verb (summarize) and resource (page). This purpose is distinct from sibling tools like scrape_page or crawl_site, making it easy for an agent to know what this tool does.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description does not explicitly state when to use this tool instead of alternatives like scrape_page or crawl_site. It does provide mode-specific guidance (auto/llm/local) and parameter details, but lacks explicit when-to-use or when-not-to-use statements relative to sibling tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Recent tool additions, removals, and schema changes observed during successful MCP inspections. Dates show when Glama detected each change.
v0.1.0click_linkcrawl_sitedownload_filequery_crawlsscrape_pagesummarize_pageEach tool has a clearly distinct purpose: full site crawling, single page scraping, summarization, link clicking, file downloading, and querying. No overlap or ambiguity.
All tools follow a consistent verb_noun pattern in snake_case (e.g., crawl_site, scrape_page, query_crawls), making the tool set predictable and easy to navigate.
With 6 tools, the server is well-scoped. It covers the essential operations for a site crawler without unnecessary bloat or missing functionality.
The tool set covers core workflows: crawling, scraping, summarizing, interacting, downloading, and querying. Minor gaps like explicit crawl session management or update/delete operations exist, but the surface is largely complete for typical use cases.
Unclaimed servers have limited discoverability.
If you are the server author, to access and configure the admin panel.
Web search, URL content extraction to Markdown, site mapping, and recursive web crawler.
Turn any URL into clean Markdown and structured data. Scrape, crawl, search and extract.
Extract public webpages as Markdown, map sites, and run bounded crawl jobs. API key required.
Scrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/coolaigit/site-crawler-mcp'
If you have feedback or need assistance with the MCP directory API, please join our Discord server
