site-crawler-mcp

A site-wide crawler MCP server built on crawl4ai (Apache-2.0). It crawls every internal link of a website like a search-engine spider (BFS), filters pages by publish time / title / URL, and can operate on crawled pages (summarize, click links, download files). Results are returned as JSON and persisted to SQLite for reuse by any project.

中文简介：一个「像谷歌爬虫一样」的全站内链爬虫 MCP。基于 crawl4ai 自研封装， 支持 BFS 全站遍历、时间/标题/URL 筛选、页面操作（LLM 概括/点击链接/下载文件）、 结果 JSON 返回 + SQLite 持久化。注册到 Reasonix / Claude Desktop / Cursor 等任意 MCP 客户端即可全局复用。

Features

• ✅ Site-wide BFS crawl — traverse all internal links (max_depth / max_pages controllable)

• ✅ Time filter — extract publish time from page meta / JSON-LD / URL first, fall back to crawl time when unavailable (result tagged with time_source)

• ✅ Title filter — include / exclude by title keyword (case-insensitive)

• ✅ URL pattern & domain filters — glob/regex URL matching, same-domain restriction

• ✅ Polite crawling — respects robots.txt and rate-limits by default (toggleable)

• ✅ Page operations — LLM summarize (LiteLLM: DeepSeek / GLM / OpenAI…, local fallback), click a specific link (CSS selector or link text), download files

• ✅ SQLite persistence — query_crawls tool to reuse crawled data in later projects

Related MCP server: Spider MCP Server

Tools

Requirements

• Python ≥ 3.12 (tested on 3.12.13)

• uv recommended (optional — plain pip works too)

• Playwright browsers: python -m playwright install chromium (or point PLAYWRIGHT_BROWSERS_PATH at an existing browser install)

Install & Register

# 1. Create environment & install
uv venv .venv --python 3.12
uv pip install --python .venv\Scripts\python.exe crawl4ai "mcp>=1.2,<2"
uv pip install --python .venv\Scripts\python.exe -e .

# 2. Install browser (once)
.venv\Scripts\python.exe -m playwright install chromium

# 3. Register as MCP server (example for Reasonix config.toml)
[[plugins]]
name    = "site-crawler-mcp"
type    = "stdio"
command = "C:\\path\\to\\site-crawler-mcp\\.venv\\Scripts\\python.exe"
args    = ["-m", "site_crawler_mcp.server"]

For Claude Desktop / Cursor, add the same command/args under mcpServers in their config files.

Quick Start (Python API)

import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

async def main():
    params = StdioServerParameters(command="python", args=["-m", "site_crawler_mcp.server"])
    async with stdio_client(params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            res = await session.call_tool("crawl_site", {
                "start_url": "https://example.com",
                "max_depth": 2,
                "max_pages": 20,
                "title_contains": "Example",
            })
            print(res.content[0].text)

asyncio.run(main())

How the time filter works

crawl4ai's URL filters work on URLs only, so content-level filtering is implemented here:

1. URL-level pruning — TimeRangeFilter / TitleFilter (URL date patterns, URL keywords)

2. Content-level — after each page is fetched, publish time is extracted from <meta property="article:published_time">, JSON-LD datePublished, <time datetime>, or YYYY/MM/DD in the URL. If none found, crawl time is used (decision documented as time_source).

Compliance

• Respects robots.txt and rate limits by default to avoid stressing target sites or getting IP-banned.

• For learning / research / your own sites; please follow target sites' ToS and local laws.

License

MIT

Built on crawl4ai (Apache-2.0).