# site-crawler-mcp

A **site-wide crawler MCP server** built on [crawl4ai](https://github.com/unclecode/crawl4ai) (Apache-2.0).
It crawls every internal link of a website like a search-engine spider (BFS), filters pages by
publish time / title / URL, and can operate on crawled pages (summarize, click links, download files).
Results are returned as JSON and persisted to SQLite for reuse by any project.

> 中文简介：一个「像谷歌爬虫一样」的全站内链爬虫 MCP。基于 crawl4ai 自研封装，
> 支持 BFS 全站遍历、时间/标题/URL 筛选、页面操作（LLM 概括/点击链接/下载文件）、
> 结果 JSON 返回 + SQLite 持久化。注册到 Reasonix / Claude Desktop / Cursor 等任意
> MCP 客户端即可全局复用。

## Features

- ✅ **Site-wide BFS crawl** — traverse all internal links (max_depth / max_pages controllable)
- ✅ **Time filter** — extract publish time from page meta / JSON-LD / URL first,
  fall back to crawl time when unavailable (result tagged with `time_source`)
- ✅ **Title filter** — include / exclude by title keyword (case-insensitive)
- ✅ **URL pattern & domain filters** — glob/regex URL matching, same-domain restriction
- ✅ **Polite crawling** — respects `robots.txt` and rate-limits by default (toggleable)
- ✅ **Page operations** — LLM summarize (LiteLLM: DeepSeek / GLM / OpenAI…, local fallback),
  click a specific link (CSS selector or link text), download files
- ✅ **SQLite persistence** — `query_crawls` tool to reuse crawled data in later projects

## Tools

| Tool | Description |
|------|-------------|
| `crawl_site` | BFS site-wide crawl. Params: `start_url`, `max_depth`, `max_pages`, `published_after/before`, `title_contains/title_exclude`, `url_pattern`, `include_external`, `respect_robots`, `rate_limit` |
| `scrape_page` | Scrape a single page (markdown / title / publish time / links) |
| `summarize_page` | Summarize page content. `mode=auto` (LLM first → local fallback) / `llm` / `local`; `llm_provider` (LiteLLM format, e.g. `deepseek/deepseek-chat`), `llm_api_key_env` (default `DEEPSEEK_API_KEY`) |
| `click_link` | Click a link inside the page (`selector` CSS or `link_text`) and scrape the target page |
| `download_file` | Download page files to output dir (default `E:\Reasonix-项目\crawler-output`) |
| `query_crawls` | Query persisted crawl results from SQLite (title/URL/time filters) |

## Requirements

- Python ≥ 3.12 (tested on 3.12.13)
- [uv](https://docs.astral.sh/uv/) recommended (optional — plain `pip` works too)
- Playwright browsers: `python -m playwright install chromium`
  (or point `PLAYWRIGHT_BROWSERS_PATH` at an existing browser install)

## Install & Register

```powershell
# 1. Create environment & install
uv venv .venv --python 3.12
uv pip install --python .venv\Scripts\python.exe crawl4ai "mcp>=1.2,<2"
uv pip install --python .venv\Scripts\python.exe -e .

# 2. Install browser (once)
.venv\Scripts\python.exe -m playwright install chromium

# 3. Register as MCP server (example for Reasonix config.toml)
[[plugins]]
name    = "site-crawler-mcp"
type    = "stdio"
command = "C:\\path\\to\\site-crawler-mcp\\.venv\\Scripts\\python.exe"
args    = ["-m", "site_crawler_mcp.server"]
```

For Claude Desktop / Cursor, add the same `command`/`args` under `mcpServers` in their config files.

## Quick Start (Python API)

```python
import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

async def main():
    params = StdioServerParameters(command="python", args=["-m", "site_crawler_mcp.server"])
    async with stdio_client(params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            res = await session.call_tool("crawl_site", {
                "start_url": "https://example.com",
                "max_depth": 2,
                "max_pages": 20,
                "title_contains": "Example",
            })
            print(res.content[0].text)

asyncio.run(main())
```

## How the time filter works

crawl4ai's URL filters work on URLs only, so content-level filtering is implemented here:

1. URL-level pruning — `TimeRangeFilter` / `TitleFilter` (URL date patterns, URL keywords)
2. Content-level — after each page is fetched, publish time is extracted from
   `<meta property="article:published_time">`, JSON-LD `datePublished`, `<time datetime>`,
   or `YYYY/MM/DD` in the URL. If none found, crawl time is used (decision documented as `time_source`).

## Compliance

- Respects `robots.txt` and rate limits by default to avoid stressing target sites or getting IP-banned.
- For learning / research / your own sites; please follow target sites' ToS and local laws.

## License

[MIT](LICENSE)

Built on [crawl4ai](https://github.com/unclecode/crawl4ai) (Apache-2.0).