scrape-mcp
You can use this MCP server as a token-efficient, anti-bot-aware web scraping and structured extraction toolkit with login/session management.
Fetch individual webpages and return compact content with configurable token budgets, markdown/text/json formats, link policies, caching, deadlines, proxies, headers/cookies, and debug diagnostics.
Extract structured data using CSS-selector schemas, including text/html/attr/count/list/object/JSON-LD fields, regex, URL absolutization, type conversion, and required-field validation.
Batch-fetch multiple URLs concurrently and receive per-URL results and progress updates.
Crawl pages either by link depth/limit or by paginated "next page" selectors.
Perform manual browser login and persist sessions, plus list, export, and import cookies for authenticated sites.
Handle anti-bot protections through tiered L1/L2 fetching, report blocked/login-required states, support PDF/plain-text fetching, and use proxies, caching, and site recipes.
Cited as a representative login-walled site: web_fetch detects the login wall and reports login_required, and the login tool opens a headed browser for a one-time manual sign-in, persisting the session to disk so later fetches of Zhihu pages automatically include the stored cookies and return the compact article body.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@scrape-mcpscrape this article and give me the key points: https://example.com/news/ai-trends"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
scrape-mcp
⚠️ 这是 scrape-mcp 的官方源仓库(upstream)。 若在别处看到同名/相似项目并带不同作者署名,即为转载或派生,非原作者发布。
反爬感知抓取 + 极省 Token 内容精简的 MCP Server —— 让 AI 读真实网页时,只花 1%~5% 的 token 就能拿到干净、真实、免登录的正文。
自动化分三层应对:默认 curl_cffi(伪装 TLS/JA3 + HTTP2)低成本打头阵,命中 Cloudflare/验证码/WAF 才升级 Playwright 真浏览器渲染;遇到"登录才给内容"的站,先弹窗手动登录一次、登录态落盘,之后自动带上、过期会提示重登。
简体中文 | English
⚡ 60 秒入门
方式 A:一键安装(推荐,零配置)
pip install mcp-scrape
# 或从源码:cd scrape-mcp && pip install -e .然后本机任意目录即可:
scrape-mcp --version # 确认装好(应输出 0.x)
claude # 在支持 MCP 的客户端里把它当服务接入不再需要 clone、手动设
PYTHONPATH或python -m;scrape-mcp命令在任意目录可用, 默认 stdio 直接被 MCP 客户端接管。想用 HTTP 直连(Postman/curl),跑scrape-mcp --transport streamable-http --port 8000。
方式 B:一键演示(拉源仓库跑 web_fetch + web_extract,看省 token 效果)
git clone https://github.com/guanyuyan/scrape-mcp.git
cd scrape-mcp
python -m venv .venv && .venv\Scripts\pip install -e .
.venv\Scripts\python scripts\demo.py预计输出(对不同站点自动选 L1/L2 分级,给出省 token 压缩比):
=== 文档站 | https://developer.mozilla.org/zh-CN/docs/Web/HTTP/CORS
ok=True tier=l1 status=200 html=260128B → tokens=3730 (约 5.7% 压缩)
标题: 跨源资源共享(CORS) - HTTP | MDN
ok=True tier=l1 data={"heading": "跨源资源共享(CORS)", "h2_count": 10, "links": 665}第 2 步:接入 MCP 客户端(Claude / Cursor 等,一条命令写好配置)
scrape-mcp init --client cursor --write # claude / vscode / json 同理默认只打印 diff 预览,确认后加 --write 落盘;重启客户端即可调用 web_fetch(url=...)。
MDN 整页约 260KB,这里只花几千 token 就拿到干净正文;拦截/登录页会自动升级或如实上报,不会把验证码当正文。
Related MCP server: foundrynet-scrape
功能说明
它解决什么:模型读网页不是在读正文,而是在读整棵 DOM(脚本、导航、版权、链接一应尽收),又贵又脏。这个工具把页面剪成"只有正文"再喂给 AI。
核心价值(三点)
极省 token —— 属性剥离 + 模板块丢弃 + 站外链接降级 + 表格转 CSV + 预算截断。实测 MDN 整页 214KB → 数百 token,压缩率约 1%。
自动分级反爬 —— 85% 的站只走最轻的 L1(快、并发高),拦截/JS 骨架页才升级 L2 真浏览器 + stealth 伪装,绝不暴力逐站上浏览器。
登录墙处理 + 如实上报 —— 判定登录/拦截时返回
login_required/blocked,引导登录而非硬撞,从不让模型把验证码当正文。
怎么用(一行)
web_fetch(url="https://xxx/article/1") → JSON:{ok, content(极省正文), token_estimate, tier, blocked?, login_required?, cached?}覆盖的站点类型:文章页 / 文档站 / 门户与列表页 / 需要 JS 渲染的 SPA / 需要登录才给正文的站(如知乎)。
真实站点实测(一键 scripts/demo.py 可复现):
站点 | 类型 | 分级 | 原始 | 抓到正文 | 说明 |
MDN CORS 文档 | 文档站 | L1 | 260KB | ~3.7k token(约 5.7%) | 纯静态,最短路径直达 |
腾讯新闻首页 | 列表/门户 | L2 | 338KB | ~1.4k token(约 1.6%) | 动态页升级浏览器渲染 |
极客邦首页 | SPA/动态 | L2 | 12KB | ~1.4k token | JS 骨架页,需渲染才有正文 |
形态:本地 MCP Server(stdio/HTTP 均可),隐私可控;支持单页 web_fetch、批量 web_batch、结构化抽取 web_extract、变更检测 web_watch、登录持久化 login,缓存默认开启。
特性
分级抓取:L1
curl_cffi→ L2 Playwright,拦截才升级,不误伤正常页面。L2 真拿到正文后按主机记 10 分钟"必须上浏览器",期间同站直接走 L2(skip_l1:sticky_l2),省掉每次注定被拦的 L1;TTL 到期或浏览器渲染失败会自动回到 L1 重新判断L1 连接复用:同一指纹模板在进程内共享连接池,省掉重复的 TLS 握手与 DNS 解析;换事件循环自动重建,进程退出或
aclose()统一释放。L2 浏览器池:同站同会话复用 context(各带独立登录态),超出上限按最久未用淘汰;同时打开的页面数受限,超出排队;空闲超过
L2_IDLE_TIMEOUT秒自动关掉浏览器进程,下次用到再启动。内容密度选根:从 article/main/div 中挑出"恰好装住正文的最紧容器",避开导航/侧栏。
compact 精简:属性剥离、模板块丢弃、站外链接降级、表格转 CSV(合并单元格自动补齐空位)、 代码块带语言标记、
figure输出图片地址与图注,压缩率可达 1~20% 数量级(如 MDN 214KB → 数百 token)。拦截如实上报:不把挑战页/空壳页当正文喂给模型,返回
blocked/block_reason。稳定错误码 + 可诊断:失败返回
error_code与retryable,debug=true还能看到分层决策、选择器命中数与 L2 交互结果。登录态持久化 + 多会话:
login工具手动登录并落盘,web_fetch自动带上;session="work"可按身份隔离登录态,session_export/session_import在机器之间搬运 cookie。非 HTML 内容:
web_fetch直接读纯文本;装[pdf]extra 后按页抽取 PDF 文本,扫描件(无文字层)如实返回empty,不伪造正文。同 URL 单飞(singleflight):同一页面被并发请求时(多租户同时抓、批量/爬取里重复链接)只真抓一次,其余等待并复用同一份响应,省出口流量也少踩目标站风控;L1 与 L2 渲染都会合并。带自定义头/凭证或
bypass_cache=true的请求可能因人而异,不参与合并。两层缓存:HTTP 响应按 TTL 落盘,过期后带
ETag/Last-Modified做条件请求;正文(compact 后)与字段抽取结果另有一层结果缓存,二次调用直接返回,bypass_cache=true同时绕过两层。web_batch并发抓取一批 URL。轻量爬取:
web_crawl支持两种模式——按depth/limit发现站内链接,或按next_selector顺序翻页(max_pages封顶、自动去重、不跳站外);静态资源自动跳过。变更检测:
web_watch每次成功抓取后留一份正文快照,下次与它对比并返回changed与增删行摘要;指定selector可只盯页面某一块(如榜单、价格区)。抓取失败不覆盖旧快照,风控恢复后仍能对比出这期间的真实变化。内置运行看板:HTTP 模式下打开
/即见请求量、L1/L2 分布、耗时分位与分桶、结果码、正文类型、L2 页位与代理池健康度;单文件 HTML,无外部依赖,每 5 秒自动刷新。总预算与降级:
web_fetch/web_extract/web_batch/web_crawl都支持deadline_ms。预算不够时按"压缩单步超时 → 跳过重试 → 不升级 L2 → 直接返回timeout"逐级降级,已拿到的结果照常返回;降级结果不写入缓存,避免后续预算充足的调用拿到残缺内容。预算同样覆盖合规检查(限流等待与 robots.txt 抓取),不够就提前返回timeout,不把时间耗在准备阶段。结构化抽取:
web_extract支持text/html/attr/count/list/object/jsonld,配regex、abs_url、convert(int/float/date)与required必填校验,缺字段直接返回missing_required;第三方包可通过入口点注册自定义字段类型。站点 recipe:把某站的渲染方式、正文选择器、抽取 schema、翻页规则写成一份 JSON 落到本地,之后同一站点免参数复用,单次调用仍可覆盖。
输出格式可选:
format=markdown|text|json,正文可按段落数组直接入库。同域并发闸门:同一站点默认最多 2 个并发请求,跨站仍可并行,避免批量抓取时把单站打穿。
Stealth 加固:Wipe
webdriver、UA 数据、Canvas/WebGL 确定性指纹噪声、时区对齐。
安装
pip install mcp-scrape # 核心:L1 抓取 + 精简 + 抽取
pip install "mcp-scrape[pdf]" # 可选:让 web_fetch 直接读 PDF# L2 渲染所需浏览器(国内可用镜像)
$env:PLAYWRIGHT_DOWNLOAD_HOST="https://cdn.npmmirror.com/binaries/playwright"
pip install "playwright>=1.49"
playwright install chromium
# 或复用系统浏览器,无需下载
$env:SCRAPE_MCP_L2_CHANNEL="chrome" # 或 msedge启动(MCP,stdio)
scrape-mcp # 默认 stdio,被 MCP 客户端接管
scrape-mcp --transport streamable-http --port 8000 # 切 HTTP 直连
scrape-mcp --version / --help # 版本 / 用法不传参默认 stdio。传
--transport/--port由 CLI 显式启动,优先级高于.env/环境变量。
环境变量(前缀 SCRAPE_MCP_)
全部配置可用环境变量(SCRAPE_MCP_ 前缀)设置,也可集中放到项目根目录 .env 文件里。
# 项目根目录执行
cp .env.example .env # 生成配置模板
# 编辑 .env 后无需改代码,直接启动即可优先级:实际进程环境变量 > .env > 默认值。.env 被 gitignore,不入库;.env.example 是随仓库提交的模板。
变量 | 默认 | 说明 |
|
| curl_cffi 指纹模板 |
|
| L1 超时(秒) |
| — | 单个代理,L1/L2 共用;配置了代理池时作为兜底 |
| — | 代理池(逗号/分号/空白分隔),按站点粘性轮询;非空时优先于 |
|
| 同一出口连续失败多少次后进入冷却 |
|
| 出口冷却时长(秒),冷却期内不再分配 |
|
| 同一主机固定走同一出口的时长(秒) |
|
| 是否启用 Playwright 升级链 |
| — | 关联浏览器:空=自带 Chromium, |
|
| 挑战页等待窗口(秒) |
|
| L2 滚动步数(触发懒加载) |
|
| L2 每步滚动间隔(毫秒) |
|
| L2 同时打开的页面上限,超出排队 |
|
| L2 常驻 context 上限(按 host + session 复用),超出淘汰最久未用 |
|
| L2 浏览器空闲回收阈值(秒), |
|
| 隐身加固时区 |
|
| 登录态/缓存放盘目录 |
|
| 站点 recipe 目录;留空用默认路径 |
|
| PDF 单次最多解析页数(超出部分不抽取, |
|
| 两层缓存 TTL(秒), |
|
| 网络异常/5xx 重试次数(429/403 不重试) |
|
| 重试退避基数(秒,指数增长) |
|
| 429/503 带 |
|
| 日志级别,JSON 行写 stderr(stdout 留给 MCP 协议) |
|
|
|
|
|
|
|
| 同一主机的并发上限( |
|
|
|
| — | 数据集导出路径(JSONL,一行一次成功抓取);留空不导出 |
|
| 默认遵循目标站 robots.txt; |
| — | 主机白名单(逗号分隔);非空时仅允许名单内主机 |
| — | 主机黑名单(逗号分隔),优先于白名单 |
|
| 全网抓取限流(每秒请求数); |
| — | HTTP 模式访问令牌;设置后除 |
| — | 多令牌 JSON: |
| — | 同格式 JSON 文件路径;适合把令牌放卷上保管(与 |
|
| 每秒请求上限(令牌桶,允许同量突发);多令牌时作为省略 |
|
| 每日请求配额(UTC 自然日,跨日清零);多令牌时作为省略 |
代理池非空时优先于 PROXY:出口按轮询分配,同一主机在 PROXY_POOL_STICKY_TTL 内固定走同一个 IP(登录态与风控都更稳);同一出口连续失败(传输异常,或整条链路走完仍被限流/风控拦截)达 PROXY_POOL_FAIL_THRESHOLD 次后进入 PROXY_POOL_COOLDOWN 秒冷却,冷却期内不再分配;整池都在冷却时回退到最早解冻的那个,而不是直接报错。单请求 proxy 参数优先级最高,且不参与池的失败计数。
HTTP 模式(Postman / curl 直连)
默认 stdio,改环境变量即可切到 HTTP,无需改代码:
$env:SCRAPE_MCP_TRANSPORT="streamable-http"
$env:SCRAPE_MCP_HTTP_PORT="8000"
$env:SCRAPE_MCP_HTTP_TOKEN="<足够长的随机字符串>" # 绑定非回环地址时必填
python -m scrape_mcp.server以 json_response=1(默认)启动,请求/响应都是普通 JSON(非事件流)。stateless 模式无需先 initialize,直接发 tools/call 即可。
访问令牌:设置 SCRAPE_MCP_HTTP_TOKEN 后,除 GET /healthz 与看板空壳 GET / 外所有请求都要带:
curl -H "Authorization: Bearer $env:SCRAPE_MCP_HTTP_TOKEN" http://127.0.0.1:8000/stats未设置令牌时服务只允许绑定 127.0.0.1 / localhost / ::1;绑定 0.0.0.0 等非回环地址会直接退出,避免无鉴权暴露。缺失或错误令牌统一返回 401(带 WWW-Authenticate),响应里不含内部路径与版本。令牌只从请求头读取,不写入日志。配置令牌后回环请求同样要携带,避免本机反向代理把外部请求伪装成 127.0.0.1 绕过鉴权。
配额:配置令牌后,该令牌的请求还受 HTTP_QPS(默认 5 次/秒)与 HTTP_DAILY_QUOTA(默认每日 10000 次,UTC 零点清零)约束,超限返回 429,响应带 Retry-After 头与 retry_after_s 字段(error 分别为 rate_limited / quota_exceeded),客户端可按建议时间退避重试。/healthz 与 /metrics 不消耗配额;未配置令牌的纯本机模式不受配额限制。
多令牌(多租户):多个调用方各拿一个令牌时,用 SCRAPE_MCP_HTTP_TOKENS(JSON)或 SCRAPE_MCP_HTTP_TOKENS_FILE(同格式文件,适合挂卷保管)一次配置多组令牌,每组独立限流,互不挤占:
[
{"name": "team-a", "token": "<随机串>", "qps": 2, "daily_quota": 2000},
{"name": "ci", "token": "<随机串>"}
]qps/daily_quota省略时继承全局HTTP_QPS/HTTP_DAILY_QUOTA,写0表示该租户不限租户名与令牌都必须唯一;配置非法(JSON 错误、缺
name/token、重复)会在启动时报错退出与单个
HTTP_TOKEN、以及HTTP_TOKENS与HTTP_TOKENS_FILE之间均互斥,同时设置会拒绝启动每个租户在
/metrics有独立的scrape_mcp_tenant_requests_total{tenant}与scrape_mcp_tenant_rejections_total{tenant,reason};429 响应不区分租户,不泄露其它调用方信息
看板:GET / 是内置运行看板(单文件 HTML,无外部依赖、无 CDN),打开即见请求量、L1/L2 分布、耗时 p50/p95/p99 与分桶、结果码、正文类型,以及 L2 页位与代理池占用,每 5 秒自动刷新。
scrape-mcp --transport streamable-http --port 8000
# 浏览器打开 http://127.0.0.1:8000/页面本身只是空壳(不含任何抓取数据),因此免鉴权;数据仍走 /stats 与 /metrics 的鉴权——配置了令牌时页面会先收到 401,填入令牌后存在浏览器 localStorage,之后由页面自动带上。
健康检查:GET /healthz 返回版本与关键开关,始终免鉴权,可用于容器探针/网关巡检。
统计:GET /stats 返回进程内计数。未配置令牌时只接受回环地址访问(其它来源返回 403);配置令牌后由 Bearer 鉴权保护,可远程读取:
curl http://127.0.0.1:8000/stats
# {"ok":true,"uptime_s":42,"requests":7,"fetches":5,"result_cache_hits":2,
# "http_cache_hits":1,"cache_hit_rate":0.286,"by_tier":{"l1":4,"l2":1},
# "by_code":{"ok":5},"by_kind":{"html":5},"tokens_returned":8120,"bytes_in":1048576,
# "latency_ms":{"window":1024,"samples":5,"p50":120,"p95":820,"p99":820,
# "buckets":{"lt_100":2,"lt_300":1,"lt_1000":1,"lt_3000":1,"lt_10000":0,"gte_10000":0},
# "by_tier":{"l1":{"samples":4,"p50":90,"p95":260},"l2":{"samples":1,"p50":820,"p95":820}}}}latency_ms 是最近 1024 次抓取的耗时分布(分位数 + 固定分桶 + 分层),不含结果缓存命中;样本有上限,长时间运行不会持续占内存。
by_tier 是实际生效的层级分布,by_code 复用工具返回的错误码(成功为 ok),cache_hit_rate 是结果缓存命中数 ÷ 总请求数。统计只记计数与字节数,不记 URL、不记页面内容,进程重启后归零。
指标:GET /metrics 输出 Prometheus 文本格式(version=0.0.4),访问限制与 /stats 完全相同,同样不含 URL 与页面内容:
curl http://127.0.0.1:8000/metrics
# scrape_mcp_fetches_total 5
# scrape_mcp_fetch_tier_total{tier="l1"} 4
# scrape_mcp_fetch_tier_total{tier="l2"} 1
# scrape_mcp_http_rejections_total{reason="unauthorized"} 0
# scrape_mcp_l2_active_pages 0指标 | 类型 | 含义 |
| counter | 总请求数 / 进入抓取链路的次数(后者含 HTTP 缓存命中、不含结果缓存命中) |
| counter | 命中层级分布( |
| counter | 按错误码 / 内容形态( |
| histogram | 抓取耗时分布(按层级,单位秒;桶 |
| counter | 两层缓存命中次数 |
| counter | 返回 token 与入站字节累计 |
| counter | HTTP 拒绝次数: |
| counter | 按租户统计的已通过请求数(只含计入配额的请求, |
| counter | 按租户统计的 429 拒绝( |
| gauge | 租户当日剩余配额(只输出设置了 |
| gauge | 当前在途 L2 渲染页数 / 排队等待页位的请求数 / 常驻浏览器上下文数 |
| gauge | 浏览器是否运行、并发页与常驻 context 上限 |
| gauge | 代理池出口总数 / 未冷却数 / 冷却中数;未配置 |
/metrics 不额外引入 prometheus_client,按文本格式直接拼装。鉴权与 /stats 相同,但不计入 QPS 与日配额,可放心让 Prometheus 按固定间隔抓取;抓取配置见 DEPLOYMENT.md。
必需项:请求头 MCP-Method、Mcp-Name;body 内 params 需带 _meta(protocolVersion + clientCapabilities)。完整示例见下。
接入 MCP 客户端(Claude / Cursor 等)
先 pip install mcp-scrape(或用项目 .venv 跑 pip install -e .),使 scrape-mcp 命令全局可用。
一条命令生成配置(推荐)
scrape-mcp init --client cursor --write # 或 claude / vscode / json默认只打印 diff 预览,加
--write才落盘;目标文件里其它服务配置原样保留--client json直接打印 JSON(不落盘),适合贴到任意客户端或自定义路径已有同名
scrape-mcp条目时不覆盖,确需更新加--force--scope global|project控制写全局还是项目内(.cursor/mcp.json、.vscode/mcp.json)--env KEY=VALUE可把SCRAPE_MCP_PROXY、SCRAPE_MCP_L2_CHANNEL等写进客户端配置--command可指定启动命令;缺省优先用已安装的scrape-mcp,找不到则退回当前解释器 +-m scrape_mcp.server
生成的结构(Claude / Cursor 用 mcpServers,VS Code 用 servers 且带 "type": "stdio"):
{
"mcpServers": {
"scrape-mcp": {
"command": "scrape-mcp",
"args": []
}
}
}若复用系统浏览器(免下载):设环境变量
SCRAPE_MCP_L2_CHANNEL为chrome或msedge。配对成功后重启客户端,即可在会话里直接调用
web_fetch/web_extract/web_batch/web_crawl。想先看工具定义(名称、说明、入参 schema):
scrape-mcp schema --tool web_fetch,或--output tools.json导出全量。没有图形界面(容器/服务器)时不必开浏览器:
scrape-mcp session import --url https://<站> --file cookies.txt直接导入浏览器扩展导出的 cookies;scrape-mcp session list查看已保存的登录态。
MCP 工具
web_fetch(url, max_tokens=4000, link_policy="internal", format="markdown", ...)→ 状态 + 极省正文link_policy:internal(站内相对链接+站外降级)/all/noneformat:markdown(默认,保留标题/列表/链接)/text(剥掉标记的纯文本)/json(content为段落数组)抓取参数:
headers/cookies(自定义请求,不走共享缓存)、timeout、bypass_cache代理:
proxy(如http://127.0.0.1:7890、socks5://127.0.0.1:1080)只对本次调用生效,L1 与升级后的 L2 走同一个出口;不传则按SCRAPE_MCP_PROXY_POOL代理池分配,未配池时沿用SCRAPE_MCP_PROXY。显式传入代理的请求不读也不写共享缓存,避免不同出口的结果混在一起总预算:
deadline_ms(毫秒,可选)限制整次调用;不够时逐级降级,见"错误码与诊断"中的deadline:*说明登录态:
session(默认default),与login时的会话名一致即可自动带上对应登录态渲染参数(仅 L2 生效):
wait_for(等 CSS 选择器)、scroll(滚动触发懒加载)、click(点"加载更多")非 HTML:PDF 返回结构化文本(装
[pdf]extra;kind="pdf"且带pages/pages_parsed),纯文本原样返回(kind="text")诊断参数:
debug=true时返回diagnostics(决策轨迹、选择器命中数、L2 交互结果),不含页面内容
web_extract(url, schema=None, ...)→ 按字段 schema 抽取结构化 JSON(可入库)schema可省略:命中站点 recipe 时直接用 recipe 里的 schema,否则返回invalid_argument字段类型:
text(默认)/html/attr(需配attr)/count/list/object/jsonld(需配path)嵌套:
list配fields返回对象数组,object在匹配节点内部继续抽取后处理:
regex+group(正则提取,未命中回落default)、abs_url(相对链接转绝对)、strip类型转换:
convert可设int/float/date(去千分位与货币符号;日期兼容 ISO 与2026年10月20日这类写法)必填校验:字段设
required: true,缺失时整次调用ok=false、error_code=missing_required,并在required_missing列出字段名jsonld用点号路径从<script type="application/ld+json">取值(多个 script /@graph自动聚合,数组自动展开、数字段视为下标)插件类型:第三方包可通过入口点注册自己的字段类型(见下文「自定义字段类型」),用法与内置类型一致
抓取/渲染参数与
web_fetch相同(不含link_policy/format);非 HTML(PDF/纯文本)返回unsupported_content,请改用web_fetch
web_batch(urls, max_tokens=3000, link_policy="internal", format="markdown", ...)→ 并发抓取一批,逐 URL 返回;单条异常不影响整批支持
headers/cookies/timeout/bypass_cache/debug/session(对该批所有 URL 生效)proxy同样对该批所有 URL 生效;缺省按代理池分配,未配池时沿用全局配置deadline_ms为整批共用的预算:耗尽后还没开始的子任务直接返回timeout,不再占用网络与浏览器客户端请求进度(传
progressToken)时,每完成一条上报已抓取 n/N 个 URL
web_crawl(url, depth=1, limit=10, same_host=True, max_tokens=2000, ...)→ 从一个入口逐页抓取,返回带mode标明模式mode=links(默认):按depth/limit发现站内链接逐层抓取;depth最大3,limit受CRAWL_MAX_PAGES约束mode=paginate:传next_selector即启用分页模式,按"下一页"链接顺序翻页,max_pages封顶(默认5),已访问 URL 自动去重,默认不跟随站外链接(same_host=false可放开)单页失败只影响该页;图片/压缩包等静态资源链接自动跳过;
session对该次所有页面生效proxy对该次所有页面生效;缺省按代理池分配,未配池时沿用全局配置deadline_ms为整次爬取的预算:耗尽后不再抓新页,已抓到的页面照常返回客户端请求进度时逐页上报(分页模式为
已翻到第 n/N 页,链接模式为已抓取 n/N 页)
web_watch(url, selector=None, reset=False, bypass_cache=True, ...)→ 对比页面自上次抓取以来的变化每次成功抓取都把正文存一份快照到
DATA_DIR/watch/,下次调用与它对比:返回changed、added_lines/removed_lines与diff预览(最多 40 行,diff_truncated标明是否截断)first_seen=true表示这是该 URL 的第一份快照(没有可比对象,故不给 diff);previous_hash/previous_chars便于人工核对selector只看页面内某一块(按命中的全部节点拼接),适合盯榜单/价格/公告位;不同selector各存一份快照,互不覆盖reset=true丢弃旧快照重建基线;比较口径忽略行尾空白与空行,避免渲染噪声误报抓取失败(拦截/超时/登录墙)不会覆盖快照,恢复后仍能对比出期间变化;
bypass_cache默认true,保证拿到的是最新内容
login(url, timeout=180, session="default")→ 打开带界面浏览器手动登录并持久化登录态多账号场景用不同
session名分别登录,web_fetch时传同一个名字即可,互不覆盖
session_list()→ 列出本地已保存的登录态(session+host+updated_at),不含 cookie 内容session_export(url, session="default", path=None)→ 导出该站 cookie({host, session, cookies});传path时同时写入本地文件session_import(url, path, session="default", replace=False)→ 从本地文件导入 cookie兼容
session_export产物、Playwrightstorage_state、裸 cookie 数组,以及浏览器扩展导出的 Netscapecookies.txt默认按
(name, domain, path)合并(同名以导入值为准),replace=true整体替换;只读本地文件,不发网络请求
返回为 JSON 字符串(落在 content[0].text,请解析文本,勿依赖 structured_content)。
失败时带稳定 error_code 与 retryable;传输失败/5xx 会自动指数退避重试,attempts 为实际尝试次数;L2 交互未命中(如选择器不存在)只在 notes 中提示,不判失败。
错误码与诊断
error_code 是协议的一部分(只增不改),retryable 直接告诉你"过一会儿重试是否有意义":
error_code | retryable | 含义 / 下一步 |
| ✅ | 网络层问题,稍后重试或检查代理 |
| ✅ | 目标服务端异常,可退避重试 |
| ❌ | 404/410 等,页面本身无效 |
| ❌ | 被反爬拦截(Cloudflare/WAF/限流),换时段或人工确认 |
| ❌ | 页面确有登录入口(密码框/登录字样),先调用 |
| ❌ | 黑白名单或 robots.txt 拒绝 |
| ❌ | 必填字段缺失,见 |
| ❌ | 内容形态不支持当前工具(如对 PDF 用 |
| ❌ | 参数问题 |
失败时默认带精简 diagnostics.decisions(如 ["retry:5xx", "escalate_l2:cloudflare", "l2_ok"]);
走粘性升级时首条是 skip_l1:sticky_l2,浏览器这次没跑起来则会看到 sticky_l2_failed:fallback_l1。
传了 deadline_ms 且触发降级时,会看到 deadline:skip_retry(预算不够再等一次退避)、
deadline:skip_l2(不够启动浏览器,返回 L1 结果)或 deadline:exceeded(没来得及发起请求)。
传 debug=true 会展开更多计数:
{"diagnostics": {
"decisions": ["escalate_l2:shell", "l2_ok"],
"attempts": 1,
"html_bytes": 348210,
"root": "div",
"selector_hits": {"article": 0, "main": 1, "div": 812},
"content_chars": 5820,
"l2": {"wait_for_hit": null, "clicked": null, "scroll_rounds": 0, "error": null}
}}429/503 带 Retry-After 时返回 retry_after_s;配置 RETRY_AFTER_MAX_WAIT 后可在上限内自动补一次重试。
web_extract 示例
schema='{"fields": {
"title": {"selector": "h1", "required": true},
"price": {"selector": ".price", "regex": "¥([\\d,.]+)", "group": 1,
"convert": "float", "required": true},
"published": {"selector": "time", "type": "attr", "attr": "datetime", "convert": "date"},
"main_link": {"selector": "a", "type": "attr", "attr": "href", "abs_url": true},
"tag_count": {"selector": ".tag", "type": "count"},
"tags": {"selector": ".tag", "type": "list", "list_key": "text"},
"author": {"type": "jsonld", "path": "author.name"},
"items": {"selector": ".item", "type": "list", "fields": {
"name": {"selector": ".name"},
"url": {"selector": "a", "type": "attr", "attr": "href", "abs_url": true}
}}
}}'
# → {"ok":true, "data": {"title":"…","price":299.0,"published":"2026-10-20",
# "main_link":"https://…","tag_count":3,"tags":["a","b","c"],
# "author":"…","items":[{"name":"…","url":"https://…"}]},
# "missing":[], "required_missing":[]}必填字段缺失时返回 ok=false + error_code=missing_required,required_missing 里是缺失的字段名:
{"ok": false, "error_code": "missing_required", "error": "必填字段缺失: title, price",
"retryable": false, "required_missing": ["title", "price"], "data": {"title": null, ...}}站点 recipe
把"这个站怎么抓、取哪块、抽哪些字段、怎么翻页"写成一份 JSON 放进 RECIPES_DIR
(默认 ~/.scrape_mcp/recipes),之后同一站点免参数复用。文件名即主机名,
也可用 host 字段覆盖;匹配按最长后缀(m.example.com 能命中 example.com),
单次调用的显式参数优先于 recipe。
~/.scrape_mcp/recipes/example.com.json:
{
"host": "example.com",
"wait_for": "main article",
"scroll": true,
"root_selector": "main article",
"extract": {
"fields": {
"title": {"selector": "h1", "required": true},
"author": {"type": "jsonld", "path": "author.name"},
"published": {"selector": "time", "type": "attr", "attr": "datetime", "convert": "date"},
"items": {
"selector": ".item",
"type": "list",
"fields": {
"name": {"selector": ".name"},
"url": {"selector": "a", "type": "attr", "attr": "href", "abs_url": true}
}
}
}
},
"crawl": {"next_selector": "a.next", "max_pages": 5}
}字段 | 作用 | 生效范围 |
| 匹配的主机(缺省用文件名) | 全部 |
| L2 渲染交互,缺省则用调用参数 |
|
| 指定正文根,命中失败自动回落密度选根 | 同上 |
|
|
|
| 分页翻页规则, |
|
| 备注,仅供人读 | — |
改完 JSON 无需重启:按文件 mtime 自动重载。写坏的 JSON 会被跳过,不影响其他 recipe。
开箱示例(真实站点)
examples/recipes/ 里有八份可直接使用的 recipe,复制到 RECIPES_DIR 即可:
mkdir -p ~/.scrape_mcp/recipes
cp examples/recipes/*.json ~/.scrape_mcp/recipes/文件 | 站点 | 演示的 recipe 能力 |
| Python 官方文档 |
|
| GitHub Trending |
|
| 阮一峰的网络日志 | 正文根 |
| MDN 文档 | 正文根用 |
| FastAPI 文档 | Material for MkDocs 站:正文根 |
| Vue 中文文档 | VitePress 站:正文根 |
| 博客园首页 | 列表页 |
| 中国新闻网文章页 | 正文根 |
# 按 recipe 抽取(不传 schema)
web_extract(url="https://docs.python.org/3/library/asyncio.html")
web_extract(url="https://github.com/trending")
web_extract(url="https://www.ruanyifeng.com/blog/2026/10/weekly-issue-415.html")
web_extract(url="https://developer.mozilla.org/zh-CN/docs/Web/HTTP/Guides/CORS")
web_extract(url="https://fastapi.tiangolo.com/tutorial/")
web_extract(url="https://cn.vuejs.org/guide/introduction.html")
web_extract(url="https://www.cnblogs.com/")
web_extract(url="https://www.chinanews.com.cn/gn/2026/10-21/10718387.shtml")
# 按 recipe 翻页(next_selector 生效即进入 mode=paginate)
web_crawl(url="https://docs.python.org/3/library/asyncio.html", max_pages=3)示例选择器于 2026-10 实测通过;站点改版后失效属正常现象,改一行 selector 即可恢复。
校验与共享
写完先本地验一遍,别等运行时才发现选择器写错:
scrape-mcp recipe validate examples/recipes # 校验目录(不传路径则校验 RECIPES_DIR)
scrape-mcp recipe validate ~/my-recipe.json # 或单个文件
scrape-mcp recipe validate examples/recipes --strict # 警告也算失败,适合挂 CI
scrape-mcp recipe list # 列出已生效的 recipe:主机、文件、启用项它会指出:JSON/编码问题(含 Windows 记事本写出的 BOM)、未知字段(带「是不是 wait_for?」式近似提示)、非法主机名、选择器语法(用运行时同一个解析器验)、extract 的 schema 错误,以及同一目录里 host 重复。退出码 0(通过)/ 1(有问题)/ 2(找不到文件),可直接用于流水线。
examples/recipes/ 就是社区配方集:加一份新配方 = 放一个 <host>.json 并用 recipe validate 验过;CI 会以 --strict 校验这八份,坏配方进不来。
单次覆盖:显式参数优先于 recipe,例如 web_fetch(url="...", wait_for=".comments")。
自定义字段类型(插件)
内置七种类型覆盖大多数页面;遇到特殊结构(meta 标签、微数据、需要跨节点计算)时,可以写个插件把自己的类型加进 web_extract,用法与内置类型完全一致。
最小插件包(本地包或发布到 PyPI 都行):
# pyproject.toml
[project]
name = "scrape-mcp-meta"
dependencies = ["mcp-scrape>=0.18"]
[project.entry-points."scrape_mcp.extractors"]
meta_content = "scrape_mcp_meta:meta_content"# scrape_mcp_meta.py
def meta_content(ctx):
# ctx.nodes 选择器命中的节点列表(可能为空)
# ctx.rule 原始字段规则,可读自定义键(如 "sep")
# ctx.root 整页解析树;ctx.base_url 页面最终地址
if not ctx.nodes:
return None
return (ctx.nodes[0].attributes or {}).get("content")装好后自动发现,直接当类型用:
{"desc": {"selector": "meta[name=description]", "type": "meta_content"}}约定与边界:
名字必须是小写标识符,不能占用内置类型名;嵌套
fields只有内置object/list支持返回字符串或列表时,
regex/abs_url/strip/convert/required照常生效插件抛异常只影响该字段:记进
errors并回落到default,同页其他字段不受牵连;返回空值同样回落default插件是进程内代码,装了什么就跑什么;不想引入第三方代码就别装扩展
数据集导出
把每次成功抓取累积成 JSONL,可直接喂给 pandas / DuckDB / 向量库。默认关闭,配一个路径就生效:
$env:SCRAPE_MCP_DATASET="D:\data\scrapes.jsonl"
# 或写进 .env:SCRAPE_MCP_DATASET=~/datasets/scrapes.jsonl之后 web_fetch / web_batch / web_extract / web_crawl 每产生一条成功结果就追加一行;失败、拦截、超时都不记,免得噪声污染数据集:
{"v": 1, "ts": "2026-10-29T12:00:00+00:00", "tool": "fetch", "url": "https://example.com/a", "final_url": "https://example.com/a", "status": 200, "tier": "l1", "title": "示例", "tokens": 1234, "chars": 5678, "content": "正文……", "error_code": null}web_extract 的结果额外带 data(字段抽取结果)。查看与转换:
scrape-mcp dataset stats # 条数、按工具分布、时间跨度、平均 token/字符
scrape-mcp dataset stats D:\data\scrapes.jsonl # 也可显式指定文件
scrape-mcp dataset to-parquet # 生成同目录 .parquet(需 mcp-scrape[dataset])边界:
只记成功结果;同一 URL 重复抓取会重复记录,去重交给下游(
url+ts足够)写盘失败只记 warning,不影响抓取本身;导出是旁路能力
文件是持续追加,长期运行请自行按天轮转(如
scrapes-2026-10-29.jsonl)
Python SDK
不接 MCP 也能直接用同一套实现(参数、返回值、错误码与工具完全一致):
import asyncio
from scrape_mcp.sdk import aclose, batch, crawl, extract, fetch
async def main() -> None:
page = await fetch("https://docs.python.org/3/library/asyncio.html", max_tokens=2000)
print(page["ok"], page["token_estimate"], page["tier"])
# 按字段抽取;不传 schema 时套用该站 recipe
data = await extract(
"https://docs.python.org/3/library/asyncio.html",
{
"fields": {
"title": {"selector": "h1"},
"sections": {"selector": "section", "type": "count"},
}
},
)
print(data["data"])
pages = await crawl(
"https://docs.python.org/3/library/asyncio.html",
next_selector='div.related a[accesskey="N"]',
max_pages=3,
)
print(pages["mode"], pages["pages_fetched"])
many = await batch(["https://example.com", "https://example.org"])
print(many["ok_count"], "/", many["total"])
await aclose() # 触发过 L2 时释放浏览器子进程;不调用也能跑,只是退出时可能有清理告警
asyncio.run(main())返回的就是 MCP 工具的 JSON 载荷(已解析为 dict),失败同样带
error_code/retryableSDK 放在
scrape_mcp.sdk,是为了不和内部scrape_mcp.extract子包重名
示例脚本
examples/ 下按场景提供 5 个可直接运行的脚本(默认参数就能跑通,需要代理时先设 SCRAPE_MCP_PROXY):
脚本 | 场景 | 常用参数 |
长文压成 token 预算内的正文 |
| |
列表页抽结构化数据(不取整页正文) |
| |
按"下一页"翻页并合并成一份 markdown |
| |
PDF 直接读成文本(需 |
| |
批量巡检,输出层级/状态/token/错误码,失败即非零退出 | 位置参数传 URL |
python examples/01_fetch_article.py https://ruanyifeng.com/blog/ --max-tokens 1200
python examples/02_extract_list.py --csv trending.csv
python examples/03_crawl_docs.py --max-pages 5 --out docs.md另有 examples/example_client.py(MCP 客户端调用示例)与 examples/recipes/*.json(站点 recipe)。
基准
两类基准,口径不同:bench.py 看缓存带来的延迟收益,bench_sites.py 看真实站点上取到的内容对不对、全不全。
缓存与延迟
python scripts/bench.py # 冷/热对照
python scripts/bench.py --l2 https://news.qq.com/ # 额外看浏览器开销"冷" = bypass_cache=true(走网络 + 精简),"热" = 普通调用(命中正文缓存)。
2026-10 实测(Windows 11 / 家宽直连,rounds=5,max_tokens=4000):
页面 | 层级 | 冷 p50 | 热 p50 | 提升 | token |
example.com | L2 | 495 ms | 1.4 ms | ~360× | 330 |
docs.python.org(asyncio-task) | L1 | 681 ms | 1.0 ms | ~650× | 3588 |
L2 浏览器:进程内首次调用 ≈3.5 s(含启动),复用 p50 ≈1.8 s。 网络与目标站状态会明显影响绝对值,重点看同一台机器上的冷热对比。
真实站点质量
python scripts/bench_sites.py # 全部站点
python scripts/bench_sites.py --nightly # 夜间回归子集
python scripts/bench_sites.py --category docs,list # 按分类过滤
python scripts/bench_sites.py --json out.json --md out.md
python scripts/bench_sites.py --baseline out.json # 与上次结果对比
python scripts/bench_sites.py --nightly --baseline benchmarks/baseline.json # 与仓库基线对比
pip install trafilatura # 可选:对照第三方正文抽取器
python scripts/bench_sites.py --reference trafilatura站点集在 benchmarks/sites.json,按文档 / 博客 / 资讯 / 列表 / 前端渲染 / 登录墙与反爬 / PDF 分层,每站可配置:
expect:ok表示应取到正文;login_required/robots_disallowed等错误码表示该站应被如实拦截,而不是伪装成功;也可以写成列表(如["ok", "login_required"]),表示不同出口下结果不同、任一都算符合预期min_chars:正文字符数下限checks:正文中必须出现的稳定片段,用于衡量完整度(只给结构稳定的页面配置,首页类动态内容不配)
输出包含通过率、取正文率、拦截符合率、完整度、token、p50/p95 延迟与分层分布,可写 JSON / Markdown 留档。
--baseline 会按站点对比上次结果,并把正文字符下降超过 20% 的站点标为退化。
同一份基线还用于耗时门禁:某站耗时超过 max(基线 × 3, 基线 + 3000ms) 即判为退化、退出码非零,夜间工作流据此开 issue;两个阈值可用 --latency-factor / --latency-floor-ms 调整。
仓库里保留一份夜间子集基线 benchmarks/baseline.json,2026-10-28 随扩容重录(14/14 通过,p50 504 ms / p95 1642 ms)。它只覆盖夜间子集,用于看趋势,不代表全量站点通过率;每月挑一次夜间结果更新:下载当次 bench-sites.json,把 sites_file 改为相对路径、补上 source 字段后替换该文件。
本机实测(2026-10-28,37 站):通过 33/37,完整度 33/33,拦截符合 4/4,延迟 p50 1705 ms / p95 9011 ms。
未通过的 4 项里,3 项是本机网络不可达(Hacker News / V2EX / arXiv 超时),1 项被站点拦截(Bilibili 返回 login_required);
这些站点在 GitHub 网络的夜间任务里会重新跑,结果以 site-quality-<run_id> 工件为准。
CI 全量(2026-10-28):通过 37/37,取正文 32/33,拦截符合 4/4,完整度 33/33,延迟 p50 1456 ms / p95 4953 ms。
其中澎湃首页在 GitHub Actions 出口被站点 WAF 恒定 403(三次全量运行都是 status=403、blocked=waf、0 字),而同站的文章页 thepaper-article 正常取到 2827 字,故首页按「如实上报拦截」计入,期望值写为 ["ok", "blocked", "login_required"]:两种出口都不算退化,正文质量由文章页覆盖。
CI 夜间子集(2026-10-28,14 站):通过 14/14,取正文 13/13,拦截符合 1/1,完整度 16/16,p50 504 ms / p95 1642 ms。
站点 | 分类 | 层级 | 结果 | 字符 | token | 完整度 | 耗时 |
py-asyncio | docs | l1 | 通过 | 31049 | 7222 | 2/2 | 2001 ms |
mdn-fetch | docs | l1 | 通过 | 2639 | 1110 | 2/2 | 2975 ms |
fastapi-first-steps | docs | l1 | 通过 | 8777 | 4124 | 2/2 | 1705 ms |
vue-intro | docs | l1 | 通过 | 4833 | 3101 | 2/2 | 2479 ms |
py-json | docs | l1 | 通过 | 27368 | 6903 | 2/2 | 256 ms |
django-tutorial | docs | l1 | 通过 | 8026 | 4243 | 2/2 | 2318 ms |
rust-ownership | docs | l1 | 通过 | 24033 | 5740 | 2/2 | 1717 ms |
ruanyifeng-blog | blog | l1 | 通过 | 908 | 543 | 2/2 | 2432 ms |
cnblogs-home | blog | l1 | 通过 | 5135 | 3976 | - | 218 ms |
sspai-home | blog | l1 | 通过 | 1613 | 1285 | 2/2 | 511 ms |
solidot-home | blog | l1 | 通过 | 8862 | 7166 | 2/2 | 553 ms |
cnblogs-article | blog | l1 | 通过 | 5883 | 5267 | - | 75 ms |
meituan-tech | blog | l1 | 通过 | 10561 | 7472 | 1/1 | 1611 ms |
chinanews-home | news | l1 | 通过 | 935 | 207 | - | 149 ms |
tencent-news-home | news | l2 | 通过 | 2022 | 1378 | - | 4893 ms |
thepaper-home | news | l2 | 通过 | 2025 | 1649 | - | 11940 ms |
36kr-home | news | l2 | 通过 | 1119 | 1060 | - | 2437 ms |
thepaper-article | news | l2 | 通过 | 2829 | 3090 | 1/1 | 1520 ms |
netease-home | news | l1 | 通过 | 10485 | 7313 | 1/1 | 406 ms |
sina-home | news | l1 | 通过 | 6360 | 7549 | 1/1 | 330 ms |
github-trending | list | l1 | 通过 | 4913 | 1787 | - | 3682 ms |
hacker-news | list | l1 | timeout(本机网络) | 0 | 0 | - | 45005 ms |
v2ex-home | list | l1 | timeout(本机网络) | 0 | 0 | - | 45020 ms |
stackoverflow-questions | list | l2 | 按预期拦截 | 0 | 0 | - | 11110 ms |
douban-home | list | l1 | 通过 | 1354 | 1028 | 1/1 | 330 ms |
react-learn | spa | l1 | 通过 | 13986 | 3514 | 2/2 | 597 ms |
apple-cn-home | spa | l2 | 通过 | 198 | 90 | 2/2 | 1564 ms |
svelte-dev | spa | l2 | 通过 | 2320 | 620 | 1/1 | 5797 ms |
tailwind-home | spa | l1 | 通过 | 6240 | 1778 | 1/1 | 3773 ms |
zhihu-question | anti_bot | - | 按预期拦截 | 0 | 0 | - | 107 ms |
xiaohongshu-explore | anti_bot | - | 按预期拦截 | 0 | 0 | - | 76 ms |
weibo-home | anti_bot | - | 按预期拦截 | 0 | 0 | - | 106 ms |
bilibili-home | anti_bot | l2 | login_required(站点拦截) | 0 | 0 | - | 2919 ms |
taobao-home | anti_bot | l2 | 通过 | 5234 | 7563 | - | 2472 ms |
arxiv-attention | l1 | timeout(本机网络) | 0 | 0 | - | 45007 ms | |
w3c-dummy-pdf | l1 | 通过 | 14 | 3 | 1/1 | 1333 ms | |
pdf-orimi | l1 | 通过 | 391 | 103 | 1/1 | 9011 ms |
--reference trafilatura 的对照只统计纯正文;本工具输出保留标题、列表与链接结构,token 数不是等价口径,只看量级即可。
Docker
# 预构建镜像(GHCR:main 推 latest,v* 标签推语义化版本)
docker pull ghcr.io/guanyuyan/scrape-mcp:latest
# 或者在本地构建;带 L2 真浏览器(Cloudflare/验证码/SPA)时把 INSTALL_BROWSER 设为 1
docker build -t scrape-mcp .
docker build --build-arg INSTALL_BROWSER=1 -t scrape-mcp:l2 .
docker run -d --name scrape-mcp -p 127.0.0.1:8002:8002 \
-e SCRAPE_MCP_HTTP_TOKEN=<令牌> \
-v scrape_data:/data \
scrape-mcp
curl http://127.0.0.1:8002/healthz # {"ok":true,"version":"…"}也可以直接 docker compose up -d --build(在 .env 里设置 SCRAPE_MCP_HTTP_TOKEN)。
镜像多阶段构建,运行阶段以非 root(uid 10001)启动;
/data卷保存登录态与两层缓存容器绑定
0.0.0.0,没有令牌会拒绝启动;/healthz免鉴权,供 HEALTHCHECK 使用反向代理、容量规划、升级与威胁模型见 DEPLOYMENT.md
目录结构
benchmarks/
├── sites.json # 真实站点质量基准站点集(分层、期望结果、完整性检查项)
└── baseline.json # 夜间子集基线(趋势参照,每月更新)
src/scrape_mcp
├── server.py # MCP 入口 + 工具(web_fetch/web_extract/web_batch/web_crawl/web_watch/login/session_*)
├── http_auth.py # HTTP Bearer 鉴权中间件 + 非回环绑定校验
├── cli.py # 子命令:init(客户端配置)/ schema(导出工具定义)
├── sdk.py # Python SDK 薄封装(fetch/extract/crawl/batch/watch/aclose)
├── config.py # 环境变量配置
├── metrics.py # Prometheus 文本格式指标(/metrics 端点)
├── tokenizer.py # tiktoken 计数 + 预算截断
├── core/
│ ├── http_client.py # L1 curl_cffi + 磁盘缓存
│ ├── fetcher.py # 分级调度(L1→L2),对工具层返回 FetchOutcome
│ ├── detector.py # 拦截/骨架页判定
│ ├── browser.py # L2 Playwright + stealth
│ ├── session.py # 登录态持久化(多会话 + cookie 导入导出)
│ ├── recipes.py # 站点 recipe(按 host 最长后缀匹配、mtime 重载)
│ ├── cache.py # HTTP 响应缓存 + 正文/抽取结果缓存
│ ├── stats.py # 进程内计数(/stats 端点数据源)
│ ├── watch.py # 变更检测(正文快照 + diff 摘要)
│ ├── errors.py # 统一错误码
│ └── hostlimit.py # 同域并发闸门
└── extract/
├── compact.py # DOM 剪枝 → 极省 token 正文
├── schema.py # 结构化抽取(按字段 schema 出 JSON)
├── links.py # 链接发现(web_crawl 用)
├── pdf.py # PDF → 文本(可选依赖 pypdf)
└── render.py # markdown → 纯文本 / 分块开发
pip install -e ".[dev]"
python -m pytestAvailable Tools
9 toolsloginA
打开带界面的浏览器手动登录,并把该站登录态持久化到磁盘。
用于 web_fetch 报告 login_required=true 的站点:浏览器会以非无头方式打开目标页, 请在弹出的窗口里完成登录;检测到登录成功或超时后,登录态(cookie 等)会被保存, 之后该站的 web_fetch 会自动带上登录态。
Args: url: 目标站点任一页面 URL(按域名区分登录态)。 timeout: 等待手动登录完成的秒数,超时也会保存当前状态便于下次继续。 session: 会话名(默认 default)。需要多账号/多身份时分别用不同名字登录, web_fetch 时传同一个 session 即可,互不覆盖。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| session | No | default | |
| timeout | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does well: it discloses that the browser is non-headless (human interaction required), that the call blocks until login succeeds or the timeout elapses, that state is saved even on timeout, and that state is keyed by domain and isolated per session. It omits where the credentials land on disk and whether an existing valid login is overwritten.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the one-line action summary, then the operational context, then per-argument notes. Slightly repetitive between the opening line and the second paragraph, but every clause carries information an agent needs.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need no explanation, and the description covers the rest of what matters for a side-effecting, human-in-the-loop tool: interaction model, timeout behavior, and state keying. Nothing essential to invoking it correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate, and it does: url is 'any page of the target site, keyed by domain', timeout is seconds waited with state still saved on expiry, and session is a namespace for parallel identities that web_fetch must match. That is real semantics — particularly the domain keying and the session must-match rule — not present in the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific action chain — open a headed browser, let the user log in manually, persist the resulting cookies to disk — and anchors it to a concrete sibling trigger (web_fetch returning login_required=true). An agent can distinguish this from session_list/session_export without reading any schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives an explicit activation condition: use it for sites where web_fetch reports login_required=true. It also explains the multi-account case via distinct session names. It stops short of saying when NOT to use it (e.g. checking existing state with session_list first, or that it is unnecessary when a session is already valid).
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
session_exportA
导出某站登录态里的 cookie,便于备份或在另一台机器上导入。
Args: url: 目标站点任一页面 URL(按域名匹配登录态)。 session: 会话名(默认 default)。 path: 可选本地文件路径;给出时把导出 JSON 写入该文件(目录需已存在), 不传则直接在返回里给出 cookies。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| path | No | ||
| session | No | default |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the burden — and it does disclose the key side effect: if path is given the JSON is written to a local file and the directory must already exist, otherwise cookies come back inline. It omits anything about overwriting an existing file, permission/auth requirements, or the sensitivity of the exported cookies.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Purpose is front-loaded in one sentence and the Args block is terse and useful; every line earns its place. Minor redundancy in restating the default value and the two-line wrap on path.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values needn't be spelled out, and the description's conditional (file vs inline) behavior covers the meaningful return branch. For a tool that writes a credential-bearing file with no annotations, an explicit note on overwrite behavior and the sensitivity of the exported cookies would close the remaining gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, but the description documents all three params with real meaning beyond the schema: url is any page URL of the target site and domain-matching is how the session is resolved, session names the session with a stated default, and path is optional with a directory-must-exist precondition and a defined alternative behavior when omitted.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (导出/export) and resource (cookies from a site's login session), with the scope rule '按域名匹配登录态'. It implicitly distinguishes itself from session_import by framing the output as something to back up or import elsewhere, but it never names the sibling or states the boundary explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The opening sentence gives clear use contexts: backup, or moving a session to another machine. It does not state any exclusions or explicitly point to session_import / session_list as the alternative path, so the when-not guidance is absent.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
session_importA
从本地文件导入 cookie 到某站登录态(仅本地文件,不发起网络请求)。
文件内容支持:session_export 导出的对象(含 cookies)、Playwright
storage_state、直接的 cookie 数组,或浏览器扩展导出的 Netscape cookies.txt。
Args:
url: 目标站点任一页面 URL(按域名匹配登录态)。
path: 本地文件路径(容器部署时用 docker cp 放进卷,再用容器内路径)。
session: 会话名(默认 default);与 web_fetch 使用的会话名保持一致。
replace: True 时用文件里的 cookie 整体替换该会话;默认 False 为按
(name, domain, path)合并,同名以导入值为准。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| path | Yes | ||
| replace | No | ||
| session | No | default |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations at all, the description carries the full burden and largely meets it: it discloses that no network request is made, and critically explains the destructive semantics of replace=True (wholesale replacement of the session's cookies) versus the default merge by (name, domain, path) where imported values win on conflict. Missing only edge behaviors such as error handling for malformed files or limits on file size.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded with the core action and its key constraint, followed by supported file formats and then per-argument notes. Every section is useful, though the Args block restates types already present in the schema and could be tightened slightly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 4-parameter tool with zero schema documentation and no annotations, the description covers file formats, container path handling, session naming, and merge-versus-replace semantics. An output schema exists, so unexplained return values are not a gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must supply all parameter meaning, and it does: url is domain-matched to a page, path includes a practical docker cp note for container deployments, session is named and tied to web_fetch consistency, and replace documents both its default and its exact merge semantics. This is substantially more than the schema conveys.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The opening sentence names a specific verb (导入/import) and resource (cookie → 站点登录态) and immediately bounds the scope with '仅本地文件,不发起网络请求'. This clearly separates it from sibling tools like login (interactive auth) and session_export (the inverse direction).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The definition establishes context well: it is the counterpart to session_export, it must share the session name used by web_fetch, and it explicitly takes a local file rather than performing network auth. It stops short of an explicit 'use login instead when...' routing statement, so a 4 rather than a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
session_listA
列出本地已保存的登录态(会话名 + 主机 + 更新时间),不含 cookie 内容。
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden. It usefully discloses that only locally saved sessions are returned and that cookie contents are excluded, which signals a safe metadata-only read. However, it does not cover permissions, what happens with no saved sessions, or any behavior beyond the data scope.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single front-loaded sentence with the resource and returned fields in a parenthetical, plus one scoping exclusion. No wasted text for a simple tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be explained further, and the tool has no parameters to document. The description is complete enough for correct invocation; only the absence of explicit sibling routing keeps it from a 5.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so the baseline is 4. The description adds the useful note that the output is limited to metadata, but there are no parameters whose semantics need explaining.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb (列出/list) and resource (本地已保存的登录态), and enumerates the returned fields (session name, host, update time). The explicit '不含 cookie 内容' implicitly separates it from session_export, though neither sibling is named directly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied by the zero-parameter read-only nature of listing sessions, but there is no explicit 'use this when...' or exclusion guidance pointing agents to session_export/import for cookie data. The agent must infer when to prefer this over the export/import siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
web_batchA
并发抓取一批 URL,各自走完整分级链路,返回逐个结果。
Args: urls: 目标 URL 列表(最多 batch_max_urls 个,默认 20)。 max_tokens: 每个 URL 的正文 token 预算上限。 link_policy: 同 web_fetch。 format: 同 web_fetch(markdown / text / json)。 headers: 所有 URL 共用的自定义请求头;传入时不走共享缓存。 cookies: 所有 URL 共用的自定义 cookie;传入时不走共享缓存。 timeout: 单次请求超时秒数。 deadline_ms: 整批共用的总预算(毫秒),不传表示不限。预算耗尽后还没开始的 子任务直接返回 timeout(diagnostics 为 deadline:exceeded),不再占用网络与浏览器。 bypass_cache: 跳过并绕过响应缓存。 debug: 每条结果附带 diagnostics(决策轨迹与计数,不含页面内容)。 session: 登录态会话名(默认 default);对该批所有 URL 生效。 proxy: 该批所有 URL 共用的代理;不传则按代理池分配,未配池时沿用全局配置。 显式传入代理时该批不读也不写共享缓存。 ctx: MCP 上下文,由框架注入,用于逐条上报进度(客户端请求进度时才发通知)。
| Name | Required | Description | Default |
|---|---|---|---|
| urls | Yes | ||
| debug | No | ||
| proxy | No | ||
| format | No | markdown | |
| cookies | No | ||
| headers | No | ||
| session | No | default | |
| timeout | No | ||
| max_tokens | No | ||
| deadline_ms | No | ||
| link_policy | No | internal | |
| bypass_cache | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does substantial work: it discloses concurrency, that deadline exhaustion returns per-subtask timeout with diagnostics 'deadline:exceeded' without consuming network/browser resources, that custom headers/cookies/proxy disable shared cache, and that progress is only reported when the client requests it. These are real behavioral traits beyond the schema. It omits rate-limit/concurrency-cap behavior and partial-failure semantics for the batch as a whole.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Front-loaded summary sentence followed by a per-argument list, so the reader gets purpose before detail. The argument block is long but each entry is one line and earns its place by documenting otherwise-undocumented params. Minor verbosity in the ctx entry, which is framework-injected and of little use to an agent.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return-value detail is rightly omitted, and the description still states 'returns per-URL results'. For a 12-param concurrent tool with no annotations, the description covers scoping, deadline behavior, cache effects, and session/proxy propagation adequately. It could say more about failure isolation across the batch and concurrency limits.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0% across 12 params, so the description must compensate and largely does: every parameter is explained, including non-obvious semantics like deadline_ms being a batch-wide budget, session applying to all URLs, and the cache side-effects of headers/cookies/proxy. The 'same as web_fetch' references for link_policy and format defer meaning to another tool's docs rather than stating it here, which is the main gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The opening line gives a specific verb+resource: concurrent fetch of a batch of URLs, each through the full tiered pipeline, returning per-URL results. It's clearly distinct from web_fetch (batch vs single) though it never explicitly names the boundary. Sibling differentiation is implied via 'link_policy/format 同 web_fetch' rather than stated.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied by 'batch' vs the singular web_fetch sibling, but there is no explicit 'use this when you have multiple URLs, use web_fetch for one'. No when-not guidance, no prerequisites. The cache-implication notes (headers/cookies/proxy disable shared cache) provide operational context but not selection guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
web_crawlA
从起始页发现站内链接并批量抓取,或按"下一页"顺序翻页,适合把一个栏目的前 N 篇读进来。
每页都走完整的分级抓取与合规链路(robots/黑白名单/限流/同域并发), 单页失败只影响该页,不中断整次爬取;静态资源链接(图片/压缩包等)自动跳过。
两种模式:
链接模式(默认):按 depth 分层发现站内链接,limit 控制总页数。
分页模式:传 next_selector(或站点 recipe 配置),顺序跟进"下一页",max_pages 控制页数。
Args: url: 起始页 URL。 depth: 链接模式的跳转层数。0=只抓起始页;1=再抓起始页上的链接(默认);最大 3。 limit: 链接模式最多抓取的页面总数(含起始页),受配置 crawl_max_pages 上限约束。 same_host: 只跟进同主机链接(默认 True)。 max_tokens: 每页正文 token 预算上限。 link_policy: 同 web_fetch。 next_selector: 分页模式:指向"下一页"的 CSS 选择器;有值时忽略 depth。 max_pages: 分页模式最多翻多少页(默认 5),受 crawl_max_pages 上限约束。 headers: 所有页面共用的自定义请求头;传入时不走共享缓存。 cookies: 所有页面共用的自定义 cookie;传入时不走共享缓存。 timeout: 单次请求超时秒数。 deadline_ms: 整次爬取的总预算(毫秒),不传表示不限。预算耗尽后不再抓新页, 已抓到的页面照常返回;排队中的子任务直接返回 timeout。 bypass_cache: 跳过并绕过响应缓存。 session: 登录态会话名(默认 default);对本次所有页面生效。 proxy: 本次爬取所有页面共用的代理;不传则按代理池分配,未配池时沿用全局配置。 显式传入代理时不走共享缓存。 ctx: MCP 上下文,由框架注入,用于逐页上报进度(客户端请求进度时才发通知)。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| depth | No | ||
| limit | No | ||
| proxy | No | ||
| cookies | No | ||
| headers | No | ||
| session | No | default | |
| timeout | No | ||
| max_pages | No | ||
| same_host | No | ||
| max_tokens | No | ||
| deadline_ms | No | ||
| link_policy | No | internal | |
| bypass_cache | No | ||
| next_selector | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does so well: every page goes through the full compliance chain (robots/allow-block list/rate limiting/same-host concurrency), a single page failure does not abort the crawl, static asset links are skipped automatically. It also discloses non-obvious side effects: explicit headers/cookies/proxy bypass the shared cache, and when deadline_ms is exhausted already-fetched pages are still returned while queued subtasks return timeout.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The purpose and mode split are front-loaded, then a structured Args list follows — the bulk is necessary given zero schema descriptions. The trailing ctx entry is framework plumbing that adds little for an agent, a small amount of dead weight.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 15-parameter crawler with no annotations, the description covers mode selection, failure isolation, caching, deadlines, and compliance — enough to call it correctly. Return values are covered by the existing output schema, but auth/session prerequisites and concurrency limits are only gestured at.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the Args block is doing all the work — and it documents 14 of 15 parameters with real semantics: depth range 0–3, limit bounded by crawl_max_pages, next_selector suppressing depth, max_pages default 5, session applying to all pages. It loses a point because a few details (e.g. timeout/proxy fallback nuances vs. global config) remain terse, but the compensation for a 15-param, 0%-covered schema is substantial.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The opening states a specific verb and resource: discover in-site links from a start page and batch crawl them, or page through via "next page". It further splits the tool into two named modes (link mode / pagination mode) so the agent understands the tool's two distinct behaviors. It only references a sibling indirectly ("link_policy: 同 web_fetch") rather than explicitly contrasting scope with web_fetch/web_batch, so it stops short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
"适合把一个栏目的前 N 篇读进来" gives a concrete selection scenario, and the two-mode block states which parameter selects which mode (next_selector overriding depth). There is no explicit "when not to use this / use web_fetch for a single page instead" exclusion, which keeps it at 4 rather than 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
web_extractA
抓取网页并按字段 schema 抽取结构化 JSON(字段级数据,可入库)。
适合需要"数据而非整页正文"的场景:给 URL 和字段规则,直接返回字段值, 而不是一段 compact 文本。抓取链路(分级反爬、登录态、合规)与 web_fetch 一致。
字段规则示例: {"fields": { "title": {"selector": "h1"}, "price": {"selector": ".price", "regex": "(\d+(?:.\d+)?)", "group": 1}, "main_link": {"selector": "a", "type": "attr", "attr": "href", "abs_url": true}, "link_count": {"selector": "a", "type": "count"}, "tags": {"selector": ".tag", "type": "list", "list_key": "text"}, "items": {"selector": ".item", "type": "list", "fields": { "name": {"selector": ".name"}, "url": {"selector": "a", "type": "attr", "attr": "href", "abs_url": true} }}, "author": {"selector": ".author", "type": "object", "fields": { "name": {"selector": ".name"}, "bio": {"selector": ".bio"} }} }} 字段类型:text(默认,节点归一化文本)/ html / attr(需配 attr)/ count(匹配节点数)/ list(配 fields 返回对象数组,否则按 list_key 取值)/ object(在第一个匹配节点内部继续按 fields 抽取)/ jsonld(不写 selector,用 path 从 JSON-LD 取值,如 {"type": "jsonld", "path": "author.name"})。 后处理:regex + group(正则提取,未命中回落 default)、abs_url(相对链接转绝对)、 strip(去首尾空白,默认 true)、convert(int / float / date,date 输出 ISO 8601)、 required(必填,缺失时整体 ok=false 并返回 missing_required)。
Args:
url: 目标网页完整 URL。
schema: 字段抽取规则字典,见上方示例;不传则套用该站 recipe 的 extract 规则。
headers: 自定义请求头;传入时不走共享缓存。
cookies: 自定义 cookie 字典;传入时不走共享缓存。
timeout: 单次请求超时秒数。
deadline_ms: 整次调用的总预算(毫秒),不传表示不限;降级规则同 web_fetch。
wait_for: 仅 L2 生效,等待该 CSS 选择器出现后再取内容;不传则套用站点 recipe。
scroll: 仅 L2 生效,滚动页面触发懒加载;不传则套用站点 recipe。
click: 仅 L2 生效,渲染后点击一次该 CSS 选择器;不传则套用站点 recipe。
bypass_cache: 跳过并绕过响应缓存,强制走网络。
debug: 返回 diagnostics(分层决策与 L2 交互命中情况),不含页面内容。
session: 登录态会话名(默认 default);需与 login 时使用的会话名一致。
proxy: 本次请求使用的代理(如 http://127.0.0.1:7890);不传则按代理池分配,
未配池时沿用全局配置。显式传入代理的请求不读也不写共享缓存。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| click | No | ||
| debug | No | ||
| proxy | No | ||
| schema | No | ||
| scroll | No | ||
| cookies | No | ||
| headers | No | ||
| session | No | default | |
| timeout | No | ||
| wait_for | No | ||
| deadline_ms | No | ||
| bypass_cache | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does so: it discloses cache-bypass semantics for headers/cookies/proxy, that wait_for/scroll/click only apply at L2, that debug returns diagnostics without page content, that session must match the login session name, that a missing required field yields ok=false with missing_required, and that an omitted schema falls back to the site recipe.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The purpose and the web_fetch contrast are front-loaded, followed by the DSL example and an arg list. It is long but the length is justified by a 13-param DSL tool; the only slack is some repetition of the fetch-chain equivalence and a very large example block.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return-shape explanation is not required, yet the description still adds the failure contract (ok=false, missing_required). Combined with the full param semantics and DSL specification, nothing an agent needs to call this correctly is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description must compensate for all 13 parameters – and it does, documenting each arg's meaning, defaults (session=default), and interactions (proxy allocation from pool, deadline_ms degradation rules same as web_fetch). The field-rule DSL is fully specified with types, post-processing options, and worked examples.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource ('抓取网页并按字段 schema 抽取结构化 JSON') and immediately draws the distinction from sibling web_fetch ('数据而非整页正文', '而不是一段 compact 文本'). An agent can pick this over web_fetch without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides a clear use-case condition – when you need fields rather than full page text – and notes that the fetch chain (anti-bot tiers, login state, compliance) is identical to web_fetch. It does not explicitly say when to prefer web_fetch or web_crawl instead, so the routing guidance is implied rather than spelled out.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
web_fetchA
抓取网页并返回极省 token 的紧凑正文(自动辨别反爬拦截页并如实上报)。
支持 HTML 页面、PDF(需安装 mcp-scrape[pdf])与纯文本(.txt/.md);
后两者直接返回文本内容,不做 DOM 精简。
Args:
url: 目标网页完整 URL,需带 http/https 协议头。
max_tokens: 正文 token 预算上限,超出时保留头尾、省略中段。
link_policy: 链接处理策略。internal=站内链接保留为相对路径、站外降级为纯文本(默认,最省);
all=站内外链接全保留;none=全部降级为纯文本。
format: 输出格式。markdown=保留标题/列表/链接标记(默认);
text=剥掉 markdown 标记的纯文本;json=按行切分的块数组(content 为字符串列表)。
headers: 自定义请求头,如 {"Referer": "...", "User-Agent": "..."};传入时不走共享缓存。
cookies: 自定义 cookie 字典;传入时不走共享缓存。
timeout: 单次请求超时秒数(默认取配置 request_timeout)。
deadline_ms: 整次调用的总预算(毫秒),不传表示不限。预算不够时逐级降级:
压缩单步超时 → 跳过重试(diagnostics 里是 deadline:skip_retry)→
不升级 L2(deadline:skip_l2)→ 未开始就耗尽时直接返回 timeout。
预算内拿到的正文照常返回,但降级结果不会写入结果缓存。
wait_for: 仅 L2 生效,等待该 CSS 选择器出现后再取内容(适合 SPA);不传则套用站点 recipe。
scroll: 仅 L2 生效,滚动页面触发懒加载/无限滚动;不传则套用站点 recipe。
click: 仅 L2 生效,渲染后点击一次该 CSS 选择器(如"加载更多");不传则套用站点 recipe。
bypass_cache: 跳过并绕过响应缓存,强制走网络。
debug: 返回 diagnostics(分层决策、L2 交互命中情况、正文根与候选命中数),
只含计数与决策、不含页面内容;失败时默认也会带精简版决策轨迹。
session: 登录态会话名(默认 default);需与 login 时使用的会话名一致,多身份互不干扰。
proxy: 本次请求使用的代理(如 http://127.0.0.1:7890、socks5://127.0.0.1:1080),
L1 与升级后的 L2 都走该出口;不传则按代理池 SCRAPE_MCP_PROXY_POOL 分配,
未配池时沿用 SCRAPE_MCP_PROXY。
显式传入代理的请求不读也不写共享缓存,避免把不同出口的结果混在一起。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| click | No | ||
| debug | No | ||
| proxy | No | ||
| format | No | markdown | |
| scroll | No | ||
| cookies | No | ||
| headers | No | ||
| session | No | default | |
| timeout | No | ||
| wait_for | No | ||
| max_tokens | No | ||
| deadline_ms | No | ||
| link_policy | No | internal | |
| bypass_cache | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does: it discloses the deadline degradation ladder (skip retry, skip L2, immediate timeout), that degraded results are not written to cache, that headers/cookies/proxy requests bypass and do not populate the shared cache, and that debug returns counts and decisions but no page content. It also notes failures still return a trimmed decision trace.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Purpose and the anti-bot caveat are front-loaded in one sentence, then a clean per-argument list. Dense but proportionate to 15 parameters; a few entries (proxy pool env vars) could be trimmed without losing callable information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
An output schema exists, so return values need not be explained, and the description still covers caching, degradation, sessions and proxies. The one real omission is sibling differentiation and the default rendering tier (what plain web_fetch does without wait_for/scroll/click).
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it documents all 15 parameters: enumerated values for link_policy (internal/all/none) and format (markdown/text/json), what bypass_cache and debug do, proxy format examples, and the L2-only restriction on wait_for/scroll/click with recipe fallback. This is materially richer than the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
A specific verb and resource (fetch a web page, return token-compact body) plus notable extras: anti-bot page detection reported faithfully, support for HTML/PDF/plain text. It never names the sibling fetchers (web_crawl, web_extract, web_batch), so an agent cannot distinguish which of the four to pick from the description alone.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage is implied rather than stated: wait_for is flagged as suitable for SPAs, session must match the name used at login, and L2 gating is mentioned. There is no explicit when-to-use / when-not guidance and no routing to web_crawl or web_extract, which matters given three closely related siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
web_watchA
检测网页(或页面内某个区块)自上次调用以来的变化。
每次成功抓取都会保存一份正文快照,下次调用与它对比,返回 changed 与增删行摘要, 适合"每天看一眼这个页面有没有新内容"的场景;无变化时 changed=false 且 diff 为空。 抓取失败(拦截/超时/登录墙)不会覆盖旧快照,等问题解决后仍能对比到真实变化。
Args: url: 目标网页完整 URL。 selector: 只看页面内的某个区块(CSS 选择器),如 "main .article";不传则对比整页正文。 reset: 丢弃该 URL 的旧快照并重建基线(本次返回 first_seen=true)。 bypass_cache: 默认 true,强制拿最新内容;false 时允许命中 CACHE_TTL 内的正文缓存。 session: 登录态会话名(默认 default);需与 login 时使用的会话名一致。 deadline_ms: 整次调用的总预算(毫秒),不传表示不限。 proxy: 本次请求使用的代理;不传则按代理池分配,未配池时沿用全局配置。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| proxy | No | ||
| reset | No | ||
| session | No | default | |
| selector | No | ||
| deadline_ms | No | ||
| bypass_cache | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden and does so well: it discloses that a snapshot is saved per successful fetch, that failures (blocking/timeout/login wall) do not overwrite the old snapshot, that reset rebuilds the baseline and returns first_seen=true, and how caching interacts with bypass_cache. Missing are concurrency/rate-limit expectations and what happens on the very first call for a URL, but the stateful semantics are unusually well surfaced.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Purpose and statefulness are front-loaded, then failure/snapshot behavior, then an Args block that maps one-to-one onto the schema. The Args block is long but each line carries operational meaning; there is no filler or repeated information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 7-parameter stateful tool with no annotations, the definition covers persistence model, failure handling, first-call/baseline behavior, caching, session coupling, and proxy/deadline semantics, and an output schema exists so return shape need not be restated. An agent has enough to call this correctly on the first attempt.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it documents all seven parameters in ways the schema cannot: reset returns first_seen=true, bypass_cache defaults to true and lets CACHE_TTL content be reused, session must match the name used at login, deadline_ms is a total call budget, and proxy falls back to the pool or global config. This adds genuine meaning beyond types and defaults.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: detect changes in a page (or a block within it) since the last call, with snapshot-based diffing. This is clearly a stateful watch tool rather than a plain fetch, so an agent can distinguish it from web_fetch/web_crawl in practice. It never names those alternatives explicitly, which keeps it just short of a 5.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives a concrete usage scenario ("每天看一眼这个页面有没有新内容") and the conditions that change the behavior for reset and bypass_cache. It also ties session usage to the login tool ("需与 login 时使用的会话名一致"), which is a real routing constraint. No explicit when-not-to-use or alternative-tool guidance is given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
1 tool update
v0.19.0- Added
web_watch
8 tool updates
v0.10.1- Changed
login1 field changed- added
Input schema / properties / sessionAdded value: +{ + "default": "default", + "title": "Session", + "type": "string" +}
- Added
session_export - Added
session_import - Added
session_list - Changed
web_batch9 fields changed- added
Input schema / properties / bypass_cacheAdded value: +{ + "default": false, + "title": "Bypass Cache", + "type": "boolean" +} - added
Input schema / properties / cookiesAdded value: +{ + "anyOf": [ + { + "additionalProperties": { + "type": "string" + }, + "type": "object" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Cookies" +} - added
Input schema / properties / deadline_msAdded value: +{ + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Deadline Ms" +} - added
Input schema / properties / debugAdded value: +{ + "default": false, + "title": "Debug", + "type": "boolean" +} - added
Input schema / properties / formatAdded value: +{ + "default": "markdown", + "title": "Format", + "type": "string" +} - added
Input schema / properties / headersAdded value: +{ + "anyOf": [ + { + "additionalProperties": { + "type": "string" + }, + "type": "object" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Headers" +} - added
Input schema / properties / proxyAdded value: +{ + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Proxy" +} - added
Input schema / properties / sessionAdded value: +{ + "default": "default", + "title": "Session", + "type": "string" +} - added
Input schema / properties / timeoutAdded value: +{ + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Timeout" +}
- Added
web_crawl - Changed
web_extract16 fields changed- added
Input schema / properties / bypass_cacheAdded value: +{ + "default": false, + "title": "Bypass Cache", + "type": "boolean" +} - added
Input schema / properties / clickAdded value: +{ + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Click" +} - added
Input schema / properties / cookiesAdded value: +{ + "anyOf": [ + { + "additionalProperties": { + "type": "string" + }, + "type": "object" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Cookies" +} - added
Input schema / properties / deadline_msAdded value: +{ + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Deadline Ms" +} - added
Input schema / properties / debugAdded value: +{ + "default": false, + "title": "Debug", + "type": "boolean" +} - added
Input schema / properties / headersAdded value: +{ + "anyOf": [ + { + "additionalProperties": { + "type": "string" + }, + "type": "object" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Headers" +} - added
Input schema / properties / proxyAdded value: +{ + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Proxy" +} - removed
Input schema / properties / schema / additionalPropertiesRemoved value: -true - added
Input schema / properties / schema / anyOfAdded value: +[ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } +] - added
Input schema / properties / schema / defaultAdded value: +null - removed
Input schema / properties / schema / typeRemoved value: -"object" - added
Input schema / properties / scrollAdded value: +{ + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Scroll" +} - added
Input schema / properties / sessionAdded value: +{ + "default": "default", + "title": "Session", + "type": "string" +} - added
Input schema / properties / timeoutAdded value: +{ + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Timeout" +} - added
Input schema / properties / wait_forAdded value: +{ + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Wait For" +} - changed
Input schema / requiredPrevious value: -[ - "url", - "schema" -]New value: +[ + "url" +]
- Changed
web_fetch12 fields changed- added
Input schema / properties / bypass_cacheAdded value: +{ + "default": false, + "title": "Bypass Cache", + "type": "boolean" +} - added
Input schema / properties / clickAdded value: +{ + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Click" +} - added
Input schema / properties / cookiesAdded value: +{ + "anyOf": [ + { + "additionalProperties": { + "type": "string" + }, + "type": "object" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Cookies" +} - added
Input schema / properties / deadline_msAdded value: +{ + "anyOf": [ + { + "type": "integer" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Deadline Ms" +} - added
Input schema / properties / debugAdded value: +{ + "default": false, + "title": "Debug", + "type": "boolean" +} - added
Input schema / properties / formatAdded value: +{ + "default": "markdown", + "title": "Format", + "type": "string" +} - added
Input schema / properties / headersAdded value: +{ + "anyOf": [ + { + "additionalProperties": { + "type": "string" + }, + "type": "object" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Headers" +} - added
Input schema / properties / proxyAdded value: +{ + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Proxy" +} - added
Input schema / properties / scrollAdded value: +{ + "anyOf": [ + { + "type": "boolean" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Scroll" +} - added
Input schema / properties / sessionAdded value: +{ + "default": "default", + "title": "Session", + "type": "string" +} - added
Input schema / properties / timeoutAdded value: +{ + "anyOf": [ + { + "type": "number" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Timeout" +} - added
Input schema / properties / wait_forAdded value: +{ + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Wait For" +}
1 tool update
v0.2.0- Added
web_extract
3 tool updates
v0.1.0- First observed
login - First observed
web_batch - First observed
web_fetch
TDQS
Scored across 9 tools
The five web_* tools have distinguishable roles: single page fetch (web_fetch), given-URL batch (web_batch), link-discovery/pagination crawl (web_crawl), schema-based structured extraction (web_extract), and change tracking (web_watch). The only borderline pairs are web_batch vs web_crawl (both fetch multiple pages) and web_extract vs web_fetch (extraction is a specialization of fetching), but descriptions make the intended split clear.
Nearly all tools follow a predictable prefix_verb_noun scheme with two natural groups: web_* for fetching operations and session_* for credential management. The lone 'login' breaks the pattern (should be session_login/web_login), a minor deviation that is still readable.
Nine tools is well within the ideal 3-15 range for a scraping server, and each tool earns its place by covering a distinct fetch mode or session operation. No filler or trivially redundant tools.
The surface covers the full scraping lifecycle: single fetch, batch, crawl, structured extraction, change monitoring, and session login/export/import/list. The main gap is session lifecycle cleanup — no logout/session_delete or cache/robots inspection tool — but agents can work around this, so it's a minor gap rather than a blocker.
Maintenance
Related MCP Connectors
Web scraping for AI agents. Converts URLs to clean, LLM-ready Markdown with anti-bot bypass.
Reliable web access for AI agents: smart HTTP, rotating proxies, and full-browser rendering.
Fetch any RSS/Atom feed as clean JSON or extract article text from any URL. Built for AI agents.
Clean Markdown and AI-readability scoring for any URL. Built for AI agents.
Related MCP Servers
- AlicenseAqualityCmaintenanceEnables AI agents to read web pages reliably, returning clean markdown content, hyperlinks, and metadata without navigation or ad noise.312 npmMIT

foundrynet-scrapeofficial
FlicenseNot gradedqualityFmaintenanceEnables AI agents to extract clean, structured content from any web page via a pay-per-call API or with a Forge key.-- AlicenseAqualityCmaintenanceEnables AI agents to extract clean, structured web content (articles, tables, links, visual layouts) optimized for LLM token efficiency, with fast response times and optional JavaScript support.538 npmMIT
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to search the web, read and extract content from webpages, fetch JSON from REST APIs, and collect links while bypassing anti-bot protections.22 npmISC