Skip to main content
Glama
site-crawler-mcp

site-crawler-mcp

by coolaigit

site-crawler-mcp

一个基于 crawl4ai (Apache-2.0) 构建的全站爬虫 MCP 服务器。 它像搜索引擎蜘蛛(BFS)一样爬取网站的所有内部链接,按发布时间/标题/URL 过滤页面,并可对爬取页面进行操作(摘要、点击链接、下载文件)。 结果以 JSON 返回并持久化到 SQLite,供任何项目复用。

中文简介:一个「像谷歌爬虫一样」的全站内链爬虫 MCP。基于 crawl4ai 自研封装, 支持 BFS 全站遍历、时间/标题/URL 筛选、页面操作(LLM 概括/点击链接/下载文件)、 结果 JSON 返回 + SQLite 持久化。注册到 Reasonix / Claude Desktop / Cursor 等任意 MCP 客户端即可全局复用。

功能特性

  • ✅ 全站 BFS 爬取 — 遍历所有内部链接(可控制 max_depth / max_pages)

  • ✅ 时间筛选 — 优先从页面 meta / JSON-LD / URL 提取发布时间, 不可用时回退到爬取时间(结果标记 time_source)

  • ✅ 标题筛选 — 按标题关键词包含/排除(不区分大小写)

  • ✅ URL 模式与域名筛选 — glob/regex URL 匹配,同域名限制

  • ✅ 礼貌爬取 — 默认遵守 robots.txt 和速率限制(可切换)

  • ✅ 页面操作 — LLM 摘要(LiteLLM: DeepSeek / GLM / OpenAI…,本地回退), 点击特定链接(CSS 选择器或链接文本),下载文件

  • ✅ SQLite 持久化 — query_crawls 工具可在后续项目中复用爬取数据

Related MCP server: read-website-fast

工具

工具

描述

crawl_site

BFS 全站爬取。参数:start_url、max_depth、max_pages、published_after/before、title_contains/title_exclude、url_pattern、include_external、respect_robots、rate_limit

scrape_page

抓取单个页面(markdown / 标题 / 发布时间 / 链接)

summarize_page

摘要页面内容。mode=auto(LLM 优先 → 本地回退)/ llm / local;llm_provider(LiteLLM 格式,例如 deepseek/deepseek-chat)、llm_api_key_env(默认 DEEPSEEK_API_KEY)

click_link

点击页面内的链接(selector CSS 或 link_text)并抓取目标页面

download_file

将页面文件下载到输出目录(默认 E:\Reasonix-项目\crawler-output)

query_crawls

从 SQLite 查询持久化的爬取结果(标题/URL/时间筛选)

环境要求

  • Python ≥ 3.12(已在 3.12.13 上测试)

  • 推荐使用 uv(可选 — 普通 pip 也可用)

  • Playwright 浏览器:python -m playwright install chromium (或将 PLAYWRIGHT_BROWSERS_PATH 指向现有浏览器安装路径)

安装与注册

# 1. Create environment & install
uv venv .venv --python 3.12
uv pip install --python .venv\Scripts\python.exe crawl4ai "mcp>=1.2,<2"
uv pip install --python .venv\Scripts\python.exe -e .

# 2. Install browser (once)
.venv\Scripts\python.exe -m playwright install chromium

# 3. Register as MCP server (example for Reasonix config.toml)
[[plugins]]
name    = "site-crawler-mcp"
type    = "stdio"
command = "C:\\path\\to\\site-crawler-mcp\\.venv\\Scripts\\python.exe"
args    = ["-m", "site_crawler_mcp.server"]

对于 Claude Desktop / Cursor,在其配置文件的 mcpServers 下添加相同的 command/args。

快速开始(Python API)

import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

async def main():
    params = StdioServerParameters(command="python", args=["-m", "site_crawler_mcp.server"])
    async with stdio_client(params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            res = await session.call_tool("crawl_site", {
                "start_url": "https://example.com",
                "max_depth": 2,
                "max_pages": 20,
                "title_contains": "Example",
            })
            print(res.content[0].text)

asyncio.run(main())

时间筛选的工作原理

crawl4ai 的 URL 筛选器仅作用于 URL,因此内容级别的筛选在此实现:

  1. URL 级别修剪 — TimeRangeFilter / TitleFilter(URL 日期模式、URL 关键词)

  2. 内容级别 — 每个页面获取后,从 <meta property="article:published_time">、JSON-LD datePublished、<time datetime> 或 URL 中的 YYYY/MM/DD 提取发布时间。如果未找到,则使用爬取时间(决策记录为 time_source)。

合规性

  • 默认遵守 robots.txt 和速率限制,以避免对目标网站造成压力或被 IP 封禁。

  • 用于学习/研究/您自己的网站;请遵守目标网站的服务条款和当地法律。

许可证

MIT

基于 crawl4ai (Apache-2.0) 构建。

Available Tools

6 tools
crawl_siteA

BFS 全站爬取(谷歌爬虫式):从 start_url 遍历站内所有内链。

筛选:published_after/before(ISO 时间,发布时间优先、抓取时间兜底)、 title_contains/title_exclude(标题包含)、url_pattern(glob/regex 模式)。 默认遵守 robots.txt 并按 0.5s/请求限速,可关闭。 结果 JSON 返回并持久化到 SQLite。

Inputs: start_url (required), max_depth, max_pages, rate_limit, +7 more.

download_fileC

下载页面文件(图片/文档等)到 E:\Reasonix-项目\crawler-output。

Inputs: url (required), filename.

query_crawlsC

查询已持久化的爬取结果(SQLite),供其它项目复用。

Inputs: limit, success_only, url_contains, title_contains, +2 more.

scrape_pageA

抓取单个页面,返回 markdown、标题、发布时间与链接列表,并持久化。

Inputs: url (required), respect_robots.

summarize_pageA

概括页面内容。mode: auto(LLM 优先,失败回落本地)/ llm / local。

llm_provider 如 "deepseek/deepseek-chat" 或 "openai/gpt-4o-mini"(LiteLLM 格式); llm_api_key_env 指定 API key 的环境变量名(默认 DEEPSEEK_API_KEY)。

Inputs: url (required), mode, llm_provider, llm_api_key_env.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.5/5.0

    Scored across 6 tools

    Disambiguation5/5

    Each tool has a clearly distinct purpose: full site crawling, single page scraping, summarization, link clicking, file downloading, and querying. No overlap or ambiguity.

    Naming Consistency5/5

    All tools follow a consistent verb_noun pattern in snake_case (e.g., crawl_site, scrape_page, query_crawls), making the tool set predictable and easy to navigate.

    Tool Count5/5

    With 6 tools, the server is well-scoped. It covers the essential operations for a site crawler without unnecessary bloat or missing functionality.

    Completeness4/5

    The tool set covers core workflows: crawling, scraping, summarizing, interacting, downloading, and querying. Minor gaps like explicit crawl session management or update/delete operations exist, but the surface is largely complete for typical use cases.

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      A
      maintenance
      Fetches webpages and crawls sites locally, stripping noise and converting content into clean, link-preserving Markdown with a minimal token footprint. Supports configurable crawl depth, concurrency, robots.txt compliance, and caching for efficient AI agent and LLM pipelines.
      MIT
    • A
      license
      Not graded
      quality
      A
      maintenance
      Enables AI agents and MCP clients to fetch webpages locally, strip noise, preserve links, and convert content into clean, token-efficient Markdown, with optional depth crawling, caching, and robots.txt-aware rate limiting.
      MIT
    • A
      license
      Not graded
      quality
      B
      maintenance
      Enables AI agents to crawl websites, follow internal links, render JavaScript pages, and extract clean main text and linked PDF tables into structured records. It exposes a crawl tool over stdio that returns records inline and writes a complete JSONL file.
      MIT