site-crawler-mcp
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| DEEPSEEK_API_KEY | No | Your DeepSeek API key for LLM summarization (default environment variable read by `summarize_page` tool) | |
| PLAYWRIGHT_BROWSERS_PATH | No | Path to an existing Playwright browsers installation, as an alternative to running `python -m playwright install chromium` |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
This server publishes no instructions, or was last inspected before Glama recorded them.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": false
} |
| prompts | {
"listChanged": false
} |
| resources | {
"subscribe": false,
"listChanged": false
} |
| experimental | {} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| crawl_siteA | BFS 全站爬取（谷歌爬虫式）：从 start_url 遍历站内所有内链。 筛选：published_after/before（ISO 时间，发布时间优先、抓取时间兜底）、 title_contains/title_exclude（标题包含）、url_pattern（glob/regex 模式）。 默认遵守 robots.txt 并按 0.5s/请求限速，可关闭。 结果 JSON 返回并持久化到 SQLite。 |
| scrape_pageA | 抓取单个页面，返回 markdown、标题、发布时间与链接列表，并持久化。 |
| summarize_pageA | 概括页面内容。mode: auto（LLM 优先，失败回落本地）/ llm / local。 llm_provider 如 "deepseek/deepseek-chat" 或 "openai/gpt-4o-mini"（LiteLLM 格式）； llm_api_key_env 指定 API key 的环境变量名（默认 DEEPSEEK_API_KEY）。 |
| click_linkC | 点击页面内指定链接（CSS selector 或链接文本）并抓取目标页内容。 |
| download_fileC | 下载页面文件（图片/文档等）到 E:\Reasonix-项目\crawler-output。 |
| query_crawlsC | 查询已持久化的爬取结果（SQLite），供其它项目复用。 |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 6 tools
Each tool has a clearly distinct purpose: full site crawling, single page scraping, summarization, link clicking, file downloading, and querying. No overlap or ambiguity.
All tools follow a consistent verb_noun pattern in snake_case (e.g., crawl_site, scrape_page, query_crawls), making the tool set predictable and easy to navigate.
With 6 tools, the server is well-scoped. It covers the essential operations for a site crawler without unnecessary bloat or missing functionality.
The tool set covers core workflows: crawling, scraping, summarizing, interacting, downloading, and querying. Minor gaps like explicit crawl session management or update/delete operations exist, but the surface is largely complete for typical use cases.