Skip to main content
Glama
scrape-mcp
by guanyuyan

scrape-mcp

⚠️ 这是 scrape-mcp 的官方源仓库(upstream)。 若在别处看到同名/相似项目并带不同作者署名,即为转载或派生,非原作者发布。

反爬感知抓取 + 极省 Token 内容精简的 MCP Server —— 让 AI 读真实网页时,只花 1%~5% 的 token 就能拿到干净、真实、免登录的正文。

自动化分三层应对:默认 curl_cffi(伪装 TLS/JA3 + HTTP2)低成本打头阵,命中 Cloudflare/验证码/WAF 才升级 Playwright 真浏览器渲染;遇到"登录才给内容"的站,先弹窗手动登录一次、登录态落盘,之后自动带上、过期会提示重登。

License: MIT PyPI Python Release CI

简体中文 | English


⚡ 60 秒入门

方式 A:一键安装(推荐,零配置)

pip install mcp-scrape
# 或从源码:cd scrape-mcp && pip install -e .

然后本机任意目录即可:

scrape-mcp --version          # 确认装好(应输出 0.x)
claude                        # 在支持 MCP 的客户端里把它当服务接入

不再需要 clone、手动设 PYTHONPATH 或 python -m;scrape-mcp 命令在任意目录可用, 默认 stdio 直接被 MCP 客户端接管。想用 HTTP 直连(Postman/curl),跑 scrape-mcp --transport streamable-http --port 8000。

方式 B:一键演示(拉源仓库跑 web_fetch + web_extract,看省 token 效果)

git clone https://github.com/guanyuyan/scrape-mcp.git
cd scrape-mcp
python -m venv .venv && .venv\Scripts\pip install -e .
.venv\Scripts\python scripts\demo.py

预计输出(对不同站点自动选 L1/L2 分级,给出省 token 压缩比):

=== 文档站 | https://developer.mozilla.org/zh-CN/docs/Web/HTTP/CORS
  ok=True tier=l1 status=200 html=260128B → tokens=3730 (约 5.7% 压缩)
  标题: 跨源资源共享(CORS) - HTTP | MDN
  ok=True tier=l1 data={"heading": "跨源资源共享(CORS)", "h2_count": 10, "links": 665}

第 2 步:接入 MCP 客户端(Claude / Cursor 等,一条命令写好配置)

scrape-mcp init --client cursor --write     # claude / vscode / json 同理

默认只打印 diff 预览,确认后加 --write 落盘;重启客户端即可调用 web_fetch(url=...)。

MDN 整页约 260KB,这里只花几千 token 就拿到干净正文;拦截/登录页会自动升级或如实上报,不会把验证码当正文。


Related MCP server: foundrynet-scrape

功能说明

它解决什么:模型读网页不是在读正文,而是在读整棵 DOM(脚本、导航、版权、链接一应尽收),又贵又脏。这个工具把页面剪成"只有正文"再喂给 AI。

核心价值(三点)

  1. 极省 token —— 属性剥离 + 模板块丢弃 + 站外链接降级 + 表格转 CSV + 预算截断。实测 MDN 整页 214KB → 数百 token,压缩率约 1%。

  2. 自动分级反爬 —— 85% 的站只走最轻的 L1(快、并发高),拦截/JS 骨架页才升级 L2 真浏览器 + stealth 伪装,绝不暴力逐站上浏览器。

  3. 登录墙处理 + 如实上报 —— 判定登录/拦截时返回 login_required/blocked,引导登录而非硬撞,从不让模型把验证码当正文。

怎么用(一行)

web_fetch(url="https://xxx/article/1") → JSON:{ok, content(极省正文), token_estimate, tier, blocked?, login_required?, cached?}

覆盖的站点类型:文章页 / 文档站 / 门户与列表页 / 需要 JS 渲染的 SPA / 需要登录才给正文的站(如知乎)。

真实站点实测(一键 scripts/demo.py 可复现):

站点

类型

分级

原始

抓到正文

说明

MDN CORS 文档

文档站

L1

260KB

~3.7k token(约 5.7%)

纯静态,最短路径直达

腾讯新闻首页

列表/门户

L2

338KB

~1.4k token(约 1.6%)

动态页升级浏览器渲染

极客邦首页

SPA/动态

L2

12KB

~1.4k token

JS 骨架页,需渲染才有正文

形态:本地 MCP Server(stdio/HTTP 均可),隐私可控;支持单页 web_fetch、批量 web_batch、结构化抽取 web_extract、变更检测 web_watch、定时监控 web_schedule、搜索 web_search、截图 web_screenshot、站点 URL 发现 web_map、sitemap 增量索引 web_index、异步爬取任务 web_crawl_start/status/cancel/list、登录持久化 login,缓存默认开启。


特性

  • 分级抓取:L1 curl_cffi → L2 Playwright,拦截才升级,不误伤正常页面。L2 真拿到正文后按主机记 10 分钟"必须上浏览器",期间同站直接走 L2(skip_l1:sticky_l2),省掉每次注定被拦的 L1;TTL 到期或浏览器渲染失败会自动回到 L1 重新判断

  • L1 连接复用:同一指纹模板在进程内共享连接池,省掉重复的 TLS 握手与 DNS 解析;换事件循环自动重建,进程退出或 aclose() 统一释放。

  • L2 浏览器池:同站同会话复用 context(各带独立登录态),超出上限按最久未用淘汰;同时打开的页面数受限,超出排队;空闲超过 L2_IDLE_TIMEOUT 秒自动关掉浏览器进程,下次用到再启动。

  • 内容密度选根:从 article/main/div 中挑出"恰好装住正文的最紧容器",避开导航/侧栏。

  • compact 精简:属性剥离、模板块丢弃、站外链接降级、表格转 CSV(合并单元格自动补齐空位)、 代码块带语言标记、figure 输出图片地址与图注,压缩率可达 1~20% 数量级(如 MDN 214KB → 数百 token)。

  • 拦截如实上报:不把挑战页/空壳页当正文喂给模型,返回 blocked/block_reason。

  • 稳定错误码 + 可诊断:失败返回 error_code 与 retryable,debug=true 还能看到分层决策、选择器命中数与 L2 交互结果。

  • 登录态持久化 + 多会话:login 工具手动登录并落盘,web_fetch 自动带上;session="work" 可按身份隔离登录态,session_export/session_import 在机器之间搬运 cookie。无 GUI 服务器可 headless=true 登录(截图落盘供查看/扫码),或用 qr_selector 只截二维码,也可用 cdp_url 连远程浏览器。

  • 非 HTML 内容:web_fetch 直接读纯文本;DOCX/XLSX/PPTX 走标准库抽取文字层(无需额外依赖);装 [pdf] extra 后按页抽取 PDF 文本;扫描件(无文字层)如实返回 empty,开启 [ocr] extra 后可选 OCR,不伪造正文。

  • 同 URL 单飞(singleflight):同一页面被并发请求时(多租户同时抓、批量/爬取里重复链接)只真抓一次,其余等待并复用同一份响应,省出口流量也少踩目标站风控;L1 与 L2 渲染都会合并。带自定义头/凭证或 bypass_cache=true 的请求可能因人而异,不参与合并。

  • 两层缓存:HTTP 响应按 TTL 落盘,过期后带 ETag/Last-Modified 做条件请求;正文(compact 后)与字段抽取结果另有一层结果缓存,二次调用直接返回,bypass_cache=true 同时绕过两层。web_batch 并发抓取一批 URL。

  • 轻量爬取:web_crawl 支持两种模式——按 depth/limit 发现站内链接,或按 next_selector 顺序翻页(max_pages 封顶、自动去重、不跳站外);静态资源自动跳过。页数多时改用 web_crawl_start 起后台任务,用 status 轮询、cancel 停止,进程重启后自动续跑。

  • 变更检测:web_watch 每次成功抓取后留一份正文快照,下次与它对比并返回 changed 与增删行摘要;指定 selector 可只盯页面某一块(如榜单、价格区)。抓取失败不覆盖旧快照,风控恢复后仍能对比出这期间的真实变化。

  • 定时监控与通知:web_schedule 把 web_watch / web_index 变成计划任务(任务可查、可暂停、可手动跑一次),由本机 scrape-mcp schedule daemon 按间隔执行;检测到真实变更后 POST 到你配置的 webhook。首次执行只建立基线,不发通知;抓取失败只记状态,不覆盖快照。

  • 搜索发现:web_search 把关键词变成结构化结果(标题 / URL / 摘要 / 来源引擎 / 发布时间),默认走 DuckDuckGo HTML 端点(无需 API key),也可指向自托管 SearXNG 的 JSON API 或配置 key 使用 Brave / Tavily;支持 site、time_range、engines 过滤与后端失败自动回退,结果可缓存,适合"先搜再抓"的两步流程。

  • 页面截图:web_screenshot 用真实浏览器渲染后导出 PNG 或整页 PDF,支持整页/视口、自定义尺寸、元素裁剪、设备预设(desktop/mobile)、截图前隐藏固定导航/弹窗,以及等待元素与额外延时;actions 可在截图前先点击/输入/翻页。复用 login 保存的登录态,需要登录或 JS 渲染的页面也能拍。

  • 站点索引:web_index 读 robots.txt 的 Sitemap:(回退 /sitemap.xml),递归展开 sitemap index 并解压 .gz;清单落盘后,下次返回新增 / 变更 / 未变 / 移除,可选回抓正文比哈希过滤"只改了时间戳"的假变更,或输出与 web_watch 同口径的差异摘要。

  • 站点 URL 发现:web_map 同时读 sitemap 与入口页站内链接(可选合并 web_search 结果),同一 URL 只出现一次并标出来源;支持 include/exclude 正则、search 子串过滤与 limit 截断,适合无 sitemap 的站点。

  • 异步爬取任务:web_crawl_start 立即返回 job_id,任务在后台推进并把状态逐页原子落盘到 DATA_DIR/jobs/;web_crawl_status 可读进度与部分结果、并在进程重启后自动续跑,web_crawl_cancel 停止任务,web_crawl_list 列出最近任务。同参数重复 start 幂等。

  • 内置运行看板:HTTP 模式下打开 / 即见请求量、L1/L2 分布、耗时分位与分桶、结果码、正文类型、L2 页位与代理池健康度;单文件 HTML,无外部依赖,每 5 秒自动刷新。

  • 总预算与降级:web_fetch / web_extract / web_batch / web_crawl / web_map / web_crawl_start 都支持 deadline_ms。预算不够时按"压缩单步超时 → 跳过重试 → 不升级 L2 → 直接返回 timeout"逐级降级,已拿到的结果照常返回;降级结果不写入缓存,避免后续预算充足的调用拿到残缺内容。预算同样覆盖合规检查(限流等待与 robots.txt 抓取),不够就提前返回 timeout,不把时间耗在准备阶段。

  • 结构化抽取:web_extract 支持 text/html/attr/count/list/object/jsonld,配 regex、abs_url、convert(int/float/date)与 required 必填校验,缺字段直接返回 missing_required;第三方包可通过入口点注册自定义字段类型。

  • 站点 recipe:把某站的渲染方式、正文选择器、抽取 schema、翻页规则写成一份 JSON 落到本地,之后同一站点免参数复用,单次调用仍可覆盖。

  • 输出格式可选:format=markdown|text|json|chunks;json 给段落数组,chunks 直接给 RAG 分块(固定窗口 + overlap + 标题路径),可原样入库。

  • 同域并发闸门:同一站点默认最多 2 个并发请求,跨站仍可并行,避免批量抓取时把单站打穿。

  • Stealth 加固:Wipe webdriver、UA 数据、Canvas/WebGL 确定性指纹噪声、时区对齐。

安装

pip install mcp-scrape                          # 核心:L1 抓取 + 精简 + 抽取
pip install "mcp-scrape[pdf]"                   # 可选:让 web_fetch 直接读 PDF
pip install "mcp-scrape[ocr]"                   # 可选:扫描 PDF 的 OCR(另需系统 tesseract 与语言包)
pip install "mcp-scrape[langchain]"             # 可选:LangChain 工具适配
pip install "mcp-scrape[llamaindex]"            # 可选:LlamaIndex Reader 适配
# L2 渲染所需浏览器(国内可用镜像)
$env:PLAYWRIGHT_DOWNLOAD_HOST="https://cdn.npmmirror.com/binaries/playwright"
pip install "playwright>=1.49"
playwright install chromium

# 或复用系统浏览器,无需下载
$env:SCRAPE_MCP_L2_CHANNEL="chrome"   # 或 msedge

启动(MCP,stdio)

scrape-mcp                          # 默认 stdio,被 MCP 客户端接管
scrape-mcp --transport streamable-http --port 8000   # 切 HTTP 直连
scrape-mcp --version / --help       # 版本 / 用法

不传参默认 stdio。传 --transport/--port 由 CLI 显式启动,优先级高于 .env/环境变量。

环境变量(前缀 SCRAPE_MCP_)

全部配置可用环境变量(SCRAPE_MCP_ 前缀)设置,也可集中放到项目根目录 .env 文件里。

# 项目根目录执行
cp .env.example .env    # 生成配置模板
# 编辑 .env 后无需改代码,直接启动即可

优先级:实际进程环境变量 > .env > 默认值。.env 被 gitignore,不入库;.env.example 是随仓库提交的模板。

变量

默认

说明

IMPERSONATE

chrome

curl_cffi 指纹模板

REQUEST_TIMEOUT

20

L1 超时(秒)

PROXY

—

单个代理,L1/L2 共用;配置了代理池时作为兜底

PROXY_POOL

—

代理池(逗号/分号/空白分隔),按站点粘性轮询;非空时优先于 PROXY

PROXY_POOL_FAIL_THRESHOLD

2

同一出口连续失败多少次后进入冷却

PROXY_POOL_COOLDOWN

60

出口冷却时长(秒),冷却期内不再分配

PROXY_POOL_STICKY_TTL

300

同一主机固定走同一出口的时长(秒)

PROXY_SESSIONS

—

会话绑定代理(名字=代理 条目,逗号/分号/空白分隔),如 work=http://1.2.3.4:8080;同一会话(含登录态)固定走该出口。优先级:单请求 proxy > 会话绑定 > 代理池 > PROXY

CAPTCHA_SOLVER_URL

—

可选 CAPTCHA 服务钩子,留空即关闭(默认)。配置后仅在检测到人机验证时把 URL/状态码/拦截类型转给该端点,接受其返回的 cookies 做一次重试

CAPTCHA_SOLVER_KEY

—

钩子服务的 Bearer 令牌(可选)

CAPTCHA_SOLVER_TIMEOUT

10

钩子调用超时(秒)

L2_ENABLED

true

是否启用 Playwright 升级链

L2_CHANNEL

—

关联浏览器:空=自带 Chromium,chrome/msedge=系统

L2_SETTLE_TIMEOUT

8

挑战页等待窗口(秒)

L2_SCROLL_STEPS

5

L2 滚动步数(触发懒加载)

L2_SCROLL_DELAY_MS

400

L2 每步滚动间隔(毫秒)

L2_MAX_PAGES

3

L2 同时打开的页面上限,超出排队

L2_MAX_CONTEXTS

8

L2 常驻 context 上限(按 host + session 复用),超出淘汰最久未用

L2_IDLE_TIMEOUT

120

L2 浏览器空闲回收阈值(秒),0 常驻到进程退出

REMOTE_BROWSER_CDP_URL

空

远程浏览器池的 CDP 端点(http/https/ws/wss);留空本地启动

REMOTE_BROWSER_FALLBACK

true

远程连接失败时是否回退本地浏览器渠道;false 直接报错

L2_TIMEZONE

Asia/Shanghai

隐身加固时区

DATA_DIR

~/.scrape_mcp

登录态/缓存放盘目录

RECIPES_DIR

<DATA_DIR>/recipes

站点 recipe 目录;留空用默认路径

PDF_MAX_PAGES

200

PDF 单次最多解析页数(超出部分不抽取,pages 仍返回总页数)

OCR_ENABLED

false

是否对无文字层的 PDF 启动 OCR(需 [ocr] extra + 系统 tesseract)

OCR_LANG

chi_sim+eng

tesseract 语言包组合

OCR_DPI

200

渲染页面时的 DPI(越高越准,也越慢)

OCR_MAX_PAGES

10

单次 OCR 最多识别页数

CACHE_TTL

300

两层缓存 TTL(秒),0 全关;HTTP 层过期后带 ETag/Last-Modified 做条件请求,正文/抽取结果层命中即直接返回

RETRY_TIMES

2

网络异常/5xx 重试次数(429/403 不重试)

RETRY_BACKOFF

0.5

重试退避基数(秒,指数增长)

RETRY_AFTER_MAX_WAIT

0

429/503 带 Retry-After 时最多等待秒数;0 只上报不等待

LOG_LEVEL

WARNING

日志级别,JSON 行写 stderr(stdout 留给 MCP 协议)

BATCH_MAX_CONCURRENCY

4

web_batch 并发上限

BATCH_MAX_URLS

20

web_batch 单批上限

HOST_MAX_CONCURRENCY

2

同一主机的并发上限(web_batch/web_crawl 共用)

CRAWL_MAX_PAGES

20

web_crawl 单次最多抓取页数硬上限

SEARCH_PROVIDER

duckduckgo

web_search 后端:duckduckgo(HTML 端点,无需 key)/ searxng(自托管 JSON API)/ brave / tavily(商业 API,需 key)

SEARCH_SEARXNG_URL

—

SearXNG 实例根地址,如 http://127.0.0.1:8080;provider=searxng 时必填

SEARCH_BRAVE_API_KEY

—

Brave Search API key;留空则该后端不可用,也不参与回退

SEARCH_TAVILY_API_KEY

—

Tavily API key;留空则该后端不可用,也不参与回退

SEARCH_FALLBACK

true

后端失败时是否自动回退到其他已配置的后端(duckduckgo → searxng → brave → tavily)

SEARCH_MAX_RESULTS

20

web_search 单次返回条数硬上限

LLM_BASE_URL

—

OpenAI 兼容端点根地址(如 https://api.deepseek.com/v1);留空则完全不发起 LLM 调用

LLM_API_KEY

—

LLM 端点 key(Bearer);本地兼容端点可不填

LLM_MODEL

—

模型名;与 LLM_BASE_URL 同时配置才启用 LLM 辅助抽取

LLM_TIMEOUT

60

单次 LLM 请求超时(秒)

LLM_MAX_INPUT_TOKENS

6000

送入模型的正文 token 预算(compact 后截断)

LLM_MAX_OUTPUT_TOKENS

2048

模型输出上限

DATASET

—

数据集导出路径(JSONL,一行一次成功抓取);留空不导出

RESPECT_ROBOTS

true

默认遵循目标站 robots.txt;false 忽略(仅建议自用合规场景关闭)

ALLOWED_HOSTS

—

主机白名单(逗号分隔);非空时仅允许名单内主机

DENIED_HOSTS

—

主机黑名单(逗号分隔),优先于白名单

MAX_QPS

10

全网抓取限流(每秒请求数);0 不限速

HTTP_TOKEN

—

HTTP 模式访问令牌;设置后除 /healthz 与看板空壳 / 外都需 Bearer,绑定非回环地址时必填(与多令牌互斥)

HTTP_TOKENS

—

多令牌 JSON:[{"name","token","qps","daily_quota","tools"}],每个租户独立配额与可选工具白名单;与 HTTP_TOKEN 互斥

HTTP_TOKENS_FILE

—

同格式 JSON 文件路径;适合把令牌放卷上保管(与 HTTP_TOKENS 互斥)

HTTP_QPS

5

每秒请求上限(令牌桶,允许同量突发);多令牌时作为省略 qps 的租户默认值;0 不限

HTTP_DAILY_QUOTA

10000

每日请求配额(UTC 自然日,跨日清零);多令牌时作为省略 daily_quota 的租户默认值;0 不限

CLOUD_BROKER_ENABLED

false

云端控制面 broker 开关;开启后暴露 /cloud/* 租约端点

CLOUD_BROWSER_UPSTREAMS

—

浏览器上游(逗号/分号/空白分隔的 CDP 端点,http/https/ws/wss),按租户轮询分配

CLOUD_PROXY_UPSTREAMS

—

代理上游(http/https/socks4/socks5),条目可带地区标签,如 socks5://host:port|us

CLOUD_LEASE_TTL

300

租约有效期(秒),到期自动回收

CLOUD_MAX_LEASES_PER_TENANT

2

单租户同时持有的租约上限(浏览器/代理分别计算);0 不限

CLOUD_DAILY_LEASES

0

单租户每日租约配额(UTC 自然日,浏览器/代理分别计算,与 HTTP 日配额独立);0 不限

CLOUD_BROKER_REDIS_URL

—

租约状态改存 Redis,多实例共享租约、轮询游标与配额;留空为进程内状态。键前缀复用 REDIS_PREFIX,需 mcp-scrape[redis]

JOB_BACKEND

local

异步 crawl 任务后端:local(本地 JSON)或 redis(共享状态 + 队列,需 mcp-scrape[redis])

REDIS_URL

redis://127.0.0.1:6379/0

JOB_BACKEND=redis 时的连接地址

REDIS_PREFIX

scrape_mcp

Redis 键前缀,便于多套环境共用实例

JOB_TTL

604800

任务状态在 Redis 中的保留秒数;0 不设过期

JOB_STALE_AFTER

600

worker 重启后,running 超过该秒数未更新才视为失联并重新入队

SNAPSHOT_BACKEND

local

web_watch / web_index(content_diff) 快照后端:local 或 s3(需 mcp-scrape[s3])

S3_BUCKET

空

SNAPSHOT_BACKEND=s3 时的桶名,必填

S3_ENDPOINT_URL

空

S3 兼容端点(MinIO / R2 / OSS);留空用 AWS 默认

S3_REGION

空

区域名;自建端点通常可留空

S3_PREFIX

scrape_mcp

对象键前缀,便于多套环境共用同一个桶

S3_ACCESS_KEY_ID / S3_SECRET_ACCESS_KEY

空

留空时走 boto3 默认凭据链(环境变量 / IAM 角色)

S3_PATH_STYLE

false

MinIO 等通常需要设为 true(path-style 寻址)

DATASET_BACKEND

local

数据集后端:local 用 DATASET 路径,s3 每条记录一个对象且无需路径

S3_JOB_ARTIFACTS

false

异步 crawl 进入终态后把任务 JSON 归档到 <S3_PREFIX>/jobs/<id>.json

代理池非空时优先于 PROXY:出口按轮询分配,同一主机在 PROXY_POOL_STICKY_TTL 内固定走同一个 IP(登录态与风控都更稳);同一出口连续失败(传输异常,或整条链路走完仍被限流/风控拦截)达 PROXY_POOL_FAIL_THRESHOLD 次后进入 PROXY_POOL_COOLDOWN 秒冷却,冷却期内不再分配;整池都在冷却时回退到最早解冻的那个,而不是直接报错。单请求 proxy 参数优先级最高,且不参与池的失败计数。

HTTP 模式(Postman / curl 直连)

默认 stdio,改环境变量即可切到 HTTP,无需改代码:

$env:SCRAPE_MCP_TRANSPORT="streamable-http"
$env:SCRAPE_MCP_HTTP_PORT="8000"
$env:SCRAPE_MCP_HTTP_TOKEN="<足够长的随机字符串>"   # 绑定非回环地址时必填
python -m scrape_mcp.server

以 json_response=1(默认)启动,请求/响应都是普通 JSON(非事件流)。stateless 模式无需先 initialize,直接发 tools/call 即可。

访问令牌:设置 SCRAPE_MCP_HTTP_TOKEN 后,除 GET /healthz 与看板空壳 GET / 外所有请求都要带:

curl -H "Authorization: Bearer $env:SCRAPE_MCP_HTTP_TOKEN" http://127.0.0.1:8000/stats

未设置令牌时服务只允许绑定 127.0.0.1 / localhost / ::1;绑定 0.0.0.0 等非回环地址会直接退出,避免无鉴权暴露。缺失或错误令牌统一返回 401(带 WWW-Authenticate),响应里不含内部路径与版本。令牌只从请求头读取,不写入日志。配置令牌后回环请求同样要携带,避免本机反向代理把外部请求伪装成 127.0.0.1 绕过鉴权。

配额:配置令牌后,该令牌的请求还受 HTTP_QPS(默认 5 次/秒)与 HTTP_DAILY_QUOTA(默认每日 10000 次,UTC 零点清零)约束,超限返回 429,响应带 Retry-After 头与 retry_after_s 字段(error 分别为 rate_limited / quota_exceeded),客户端可按建议时间退避重试。/healthz 与 /metrics 不消耗配额;未配置令牌的纯本机模式不受配额限制。

多令牌(多租户):多个调用方各拿一个令牌时,用 SCRAPE_MCP_HTTP_TOKENS(JSON)或 SCRAPE_MCP_HTTP_TOKENS_FILE(同格式文件,适合挂卷保管)一次配置多组令牌,每组独立限流,互不挤占:

[
  {"name": "team-a", "token": "<随机串>", "qps": 2, "daily_quota": 2000},
  {"name": "ci", "token": "<随机串>", "tools": ["web_fetch", "web_search", "web_extract"]}
]
  • qps / daily_quota 省略时继承全局 HTTP_QPS / HTTP_DAILY_QUOTA,写 0 表示该租户不限

  • tools 可选:只允许调用列出的工具,省略或写 "*" 表示全部放行;调用白名单外的工具返回 403(error=tool_forbidden,并带 tool 字段),不会执行任何抓取;批量请求中只要有一个工具越权就整批拒绝

  • 租户名与令牌都必须唯一;配置非法(JSON 错误、缺 name/token、重复、tools 非法)会在启动时报错退出

  • 与单个 HTTP_TOKEN、以及 HTTP_TOKENS 与 HTTP_TOKENS_FILE 之间均互斥,同时设置会拒绝启动

  • 每个租户在 /metrics 有独立的 scrape_mcp_tenant_requests_total{tenant}、scrape_mcp_tenant_rejections_total{tenant,reason} 与 scrape_mcp_tenant_tool_calls_total{tenant,tool};429/403 响应不区分其它租户,不泄露其它调用方信息

看板:GET / 是内置运行看板(单文件 HTML,无外部依赖、无 CDN),打开即见请求量、L1/L2 分布、耗时 p50/p95/p99 与分桶、结果码、正文类型,以及 L2 页位与代理池占用,每 5 秒自动刷新。

scrape-mcp --transport streamable-http --port 8000
# 浏览器打开 http://127.0.0.1:8000/

页面本身只是空壳(不含任何抓取数据),因此免鉴权;数据仍走 /stats 与 /metrics 的鉴权——配置了令牌时页面会先收到 401,填入令牌后存在浏览器 localStorage,之后由页面自动带上。

健康检查:GET /healthz 返回版本与关键开关,始终免鉴权,可用于容器探针/网关巡检。

统计:GET /stats 返回进程内计数。未配置令牌时只接受回环地址访问(其它来源返回 403);配置令牌后由 Bearer 鉴权保护,可远程读取:

curl http://127.0.0.1:8000/stats
# {"ok":true,"uptime_s":42,"requests":7,"fetches":5,"result_cache_hits":2,
#  "http_cache_hits":1,"cache_hit_rate":0.286,"by_tier":{"l1":4,"l2":1},
#  "by_code":{"ok":5},"by_kind":{"html":5},"tokens_returned":8120,"bytes_in":1048576,
#  "latency_ms":{"window":1024,"samples":5,"p50":120,"p95":820,"p99":820,
#   "buckets":{"lt_100":2,"lt_300":1,"lt_1000":1,"lt_3000":1,"lt_10000":0,"gte_10000":0},
#   "by_tier":{"l1":{"samples":4,"p50":90,"p95":260},"l2":{"samples":1,"p50":820,"p95":820}}}}

latency_ms 是最近 1024 次抓取的耗时分布(分位数 + 固定分桶 + 分层),不含结果缓存命中;样本有上限,长时间运行不会持续占内存。

by_tier 是实际生效的层级分布,by_code 复用工具返回的错误码(成功为 ok),cache_hit_rate 是结果缓存命中数 ÷ 总请求数。统计只记计数与字节数,不记 URL、不记页面内容,进程重启后归零。

多令牌模式下 /stats 还带 tenants:按租户的 requests、tool_calls(工具分布)、rejections(拒绝原因)与 quota_remaining(仅设置了日配额的租户);未配置令牌时为空对象。

指标:GET /metrics 输出 Prometheus 文本格式(version=0.0.4),访问限制与 /stats 完全相同,同样不含 URL 与页面内容:

curl http://127.0.0.1:8000/metrics
# scrape_mcp_fetches_total 5
# scrape_mcp_fetch_tier_total{tier="l1"} 4
# scrape_mcp_fetch_tier_total{tier="l2"} 1
# scrape_mcp_http_rejections_total{reason="unauthorized"} 0
# scrape_mcp_l2_active_pages 0

指标

类型

含义

scrape_mcp_requests_total / scrape_mcp_fetches_total

counter

总请求数 / 进入抓取链路的次数(后者含 HTTP 缓存命中、不含结果缓存命中)

scrape_mcp_fetch_tier_total{tier}

counter

命中层级分布(l1 = curl_cffi,l2 = Playwright)

scrape_mcp_fetch_code_total{code} / scrape_mcp_fetch_kind_total{kind}

counter

按错误码 / 内容形态(html/pdf/text)分布

scrape_mcp_fetch_duration_seconds

histogram

抓取耗时分布(按层级,单位秒;桶 0.1/0.3/1/3/10/+Inf),可直接算 p95 与平均耗时

scrape_mcp_result_cache_hits_total / scrape_mcp_http_cache_hits_total

counter

两层缓存命中次数

scrape_mcp_tokens_returned_total / scrape_mcp_bytes_in_total

counter

返回 token 与入站字节累计

scrape_mcp_http_rejections_total{reason}

counter

HTTP 拒绝次数:unauthorized / forbidden / tool_forbidden / rate_limited / quota_exceeded

scrape_mcp_tenant_requests_total{tenant}

counter

按租户统计的已通过请求数(只含计入配额的请求,/healthz、/metrics 不计)

scrape_mcp_tenant_rejections_total{tenant,reason}

counter

按租户统计的拒绝:429 限流/配额与 403 工具白名单(rate_limited / quota_exceeded / tool_forbidden)

scrape_mcp_tenant_tool_calls_total{tenant,tool}

counter

按租户与工具统计的已执行调用次数(通过权限与配额的 tools/call)

scrape_mcp_tenant_quota_remaining{tenant}

gauge

租户当日剩余配额(只输出设置了 daily_quota 的租户,可用来做余量告警)

scrape_mcp_l2_active_pages / scrape_mcp_l2_queued_pages / scrape_mcp_l2_contexts

gauge

当前在途 L2 渲染页数 / 排队等待页位的请求数 / 常驻浏览器上下文数

scrape_mcp_l2_browser_running / scrape_mcp_l2_max_pages / scrape_mcp_l2_max_contexts

gauge

浏览器是否运行、并发页与常驻 context 上限

scrape_mcp_proxy_pool_size / _healthy / _cooling

gauge

代理池出口总数 / 未冷却数 / 冷却中数;未配置 PROXY_POOL 时不输出这组指标

/metrics 不额外引入 prometheus_client,按文本格式直接拼装。鉴权与 /stats 相同,但不计入 QPS 与日配额,可放心让 Prometheus 按固定间隔抓取;抓取配置见 DEPLOYMENT.md。

请求关联 ID:每次工具调用都会生成一个 12 位 request_id,同时出现在返回载荷与 stderr 的 JSON 日志里(字段同名)。web_batch / web_crawl 的子任务共用外层 ID,因此一次调用只有一个 ID;结果缓存命中时载荷里是本次调用新生成的值,不会沿用首次请求的 ID。报障时提供该 ID 即可按日志定位整条链路。

链路追踪(可选):SCRAPE_MCP_OTEL_ENABLED=1 + pip install "mcp-scrape[otel]" 后,每次工具调用产生 tool.<name> span、每次抓取产生 fetch span(属性含 request_id / tier / status / code / elapsed_ms),按 OTLP/HTTP 导出到 SCRAPE_MCP_OTEL_ENDPOINT(默认 http://127.0.0.1:4318)。默认关闭;未安装可选依赖时只记一条警告并退化为空操作,不影响抓取。

必需项:请求头 MCP-Method、Mcp-Name;body 内 params 需带 _meta(protocolVersion + clientCapabilities)。完整示例见下。

接入 MCP 客户端(Claude / Cursor 等)

先 pip install mcp-scrape(或用项目 .venv 跑 pip install -e .),使 scrape-mcp 命令全局可用。

一条命令生成配置(推荐)

scrape-mcp init --client cursor --write     # 或 claude / vscode / json
  • 默认只打印 diff 预览,加 --write 才落盘;目标文件里其它服务配置原样保留

  • --client json 直接打印 JSON(不落盘),适合贴到任意客户端或自定义路径

  • 已有同名 scrape-mcp 条目时不覆盖,确需更新加 --force

  • --scope global|project 控制写全局还是项目内(.cursor/mcp.json、.vscode/mcp.json)

  • --env KEY=VALUE 可把 SCRAPE_MCP_PROXY、SCRAPE_MCP_L2_CHANNEL 等写进客户端配置

  • --command 可指定启动命令;缺省优先用已安装的 scrape-mcp,找不到则退回当前解释器 + -m scrape_mcp.server

生成的结构(Claude / Cursor 用 mcpServers,VS Code 用 servers 且带 "type": "stdio"):

{
  "mcpServers": {
    "scrape-mcp": {
      "command": "scrape-mcp",
      "args": []
    }
  }
}
  • 若复用系统浏览器(免下载):设环境变量 SCRAPE_MCP_L2_CHANNEL 为 chrome 或 msedge。

  • 配对成功后重启客户端,即可在会话里直接调用 web_fetch / web_extract / web_batch / web_crawl / web_map / web_crawl_start。

  • 想先看工具定义(名称、说明、入参 schema):scrape-mcp schema --tool web_fetch,或 --output tools.json 导出全量。

  • 没有图形界面(容器/服务器)时不必开浏览器:scrape-mcp session import --url https://<站> --file cookies.txt 直接导入浏览器扩展导出的 cookies;scrape-mcp session list 查看已保存的登录态。

MCP 工具

  • web_fetch(url, max_tokens=4000, link_policy="internal", format="markdown", ...) → 状态 + 极省正文

    • link_policy: internal(站内相对链接+站外降级)/ all / none

    • format: markdown(默认,保留标题/列表/链接)/ text(剥掉标记的纯文本)/ json(content 为段落数组)/ chunks(RAG 分块,见下)

    • format=chunks: content 为分块数组,每块含 index / text / heading_path(标题层级路径)/ start / end(在正文中的字符偏移)/ tokens;顶层另有 chunk_count,可直接写入向量库

    • chunk_size / chunk_overlap: chunks 模式的窗口与重叠字符数(默认 1200 / 200;窗口 100~8000,重叠不超过窗口一半)

    • 抓取参数:headers / cookies(自定义请求,不走共享缓存)、timeout、bypass_cache

    • 代理:proxy(如 http://127.0.0.1:7890、socks5://127.0.0.1:1080)只对本次调用生效,L1 与升级后的 L2 走同一个出口;不传则按代理池分配,未配池时沿用 SCRAPE_MCP_PROXY。显式传入代理的请求不读也不写共享缓存,避免不同出口的结果混在一起

    • 出口优先级:单请求 proxy > 会话绑定(SCRAPE_MCP_PROXY_SESSIONS)> 代理池 > 全局 PROXY

    • 地区定向:region(如 us / hk / jp)只在走代理池时生效,仅从带该地区标签的出口中选择(池条目写成 http://host:port|us);没有该地区出口时直接返回 invalid_argument,不会悄悄换出口。会话绑定了代理或显式传了 proxy 时不生效,notes 会说明实际出口

    • CAPTCHA 钩子(SCRAPE_MCP_CAPTCHA_SOLVER_URL)默认关闭且不做破解:仅在检测到人机验证时把 URL/状态码/拦截类型转给你指定的合规服务(人工值守或已获授权的 solver),按对方返回的 cookies 重试一次;失败只写 notes,不影响原有结果。使用前请确认目标站条款与当地法律,不要用于大规模绕过验证或批量注册

    • 总预算:deadline_ms(毫秒,可选)限制整次调用;不够时逐级降级,见"错误码与诊断"中的 deadline:* 说明

    • 登录态:session(默认 default),与 login 时的会话名一致即可自动带上对应登录态

    • 渲染参数(仅 L2 生效):wait_for(等 CSS 选择器)、scroll(滚动触发懒加载)、click(点"加载更多");actions(动作链,按顺序执行 click/type/press/select/wait/scroll,单个动作失败只写入 notes)

    • 非 HTML:PDF 返回结构化文本(装 [pdf] extra;kind="pdf" 且带 pages/pages_parsed);DOCX/XLSX/PPTX 直接用标准库抽取(kind 对应 docx/xlsx/pptx);纯文本原样返回(kind="text")

    • 扫描 PDF:OCR_ENABLED=1 且装了 [ocr] extra 时对无文字层页面 OCR,载荷带 ocr=true;OCR 不可用或仍无文字时返回 unsupported_content / empty

    • 诊断参数:debug=true 时返回 diagnostics(决策轨迹、选择器命中数、L2 交互结果),不含页面内容

  • web_extract(url, schema=None, ...) → 按字段 schema 抽取结构化 JSON(可入库)

    • schema 可省略:命中站点 recipe 时直接用 recipe 里的 schema,否则返回 invalid_argument

    • 字段类型:text(默认)/ html / attr(需配 attr)/ count / list / object / jsonld(需配 path)

    • 嵌套:list 配 fields 返回对象数组,object 在匹配节点内部继续抽取

    • 后处理:regex + group(正则提取,未命中回落 default)、abs_url(相对链接转绝对)、strip

    • 类型转换:convert 可设 int / float / date(去千分位与货币符号;日期兼容 ISO 与 2026年10月20日 这类写法)

    • 必填校验:字段设 required: true,缺失时整次调用 ok=false、error_code=missing_required,并在 required_missing 列出字段名

    • jsonld 用点号路径从 <script type="application/ld+json"> 取值(多个 script / @graph 自动聚合,数组自动展开、数字段视为下标)

    • 插件类型:第三方包可通过入口点注册自己的字段类型(见下文「自定义字段类型」),用法与内置类型一致

    • 抓取/渲染参数与 web_fetch 相同(不含 link_policy/format);非 HTML(PDF/纯文本)返回 unsupported_content,请改用 web_fetch

    • prompt(可选):LLM 辅助抽取。配置 LLM_BASE_URL + LLM_MODEL 后把 compact 正文交给 OpenAI 兼容模型;未配置或调用/解析失败自动回退确定性 schema,不会产生默认外部调用

    • 返回 extract_mode(llm / schema)与 model;回退原因写在 notes。字段规则可用 description 说明含义,会随 prompt 一起交给模型

    • 隐私提示:启用 LLM 意味着页面正文会发送到你配置的端点,不要把不可外发的页面交给它

  • web_batch(urls, max_tokens=3000, link_policy="internal", format="markdown", ...) → 并发抓取一批,逐 URL 返回;单条异常不影响整批

    • 支持 headers / cookies / timeout / bypass_cache / debug / session(对该批所有 URL 生效);format=chunks 与 chunk_size / chunk_overlap 同样逐条生效

    • proxy 同样对该批所有 URL 生效;缺省按代理池分配,未配池时沿用全局配置

    • region 同样对该批所有 URL 生效(如 us);只在走代理池时生效,无该地区出口时整批返回 invalid_argument

    • deadline_ms 为整批共用的预算:耗尽后还没开始的子任务直接返回 timeout,不再占用网络与浏览器

    • 客户端请求进度(传 progressToken)时,每完成一条上报 已抓取 n/N 个 URL

  • web_crawl(url, depth=1, limit=10, same_host=True, max_tokens=2000, ...) → 从一个入口逐页抓取,返回带 mode 标明模式

    • mode=links(默认):按 depth/limit 发现站内链接逐层抓取;depth 最大 3,limit 受 CRAWL_MAX_PAGES 约束

    • mode=paginate:传 next_selector 即启用分页模式,按"下一页"链接顺序翻页,max_pages 封顶(默认 5),已访问 URL 自动去重,默认不跟随站外链接(same_host=false 可放开)

    • 单页失败只影响该页;图片/压缩包等静态资源链接自动跳过;session 对该次所有页面生效

    • proxy 对该次所有页面生效;缺省按代理池分配,未配池时沿用全局配置

    • region 对该次所有页面生效(如 us);只在走代理池时生效,无该地区出口时直接返回 invalid_argument

    • deadline_ms 为整次爬取的预算:耗尽后不再抓新页,已抓到的页面照常返回

    • 客户端请求进度时逐页上报(分页模式为 已翻到第 n/N 页,链接模式为 已抓取 n/N 页)

  • web_crawl_start(url, depth=1, limit=10, ..., next_selector=None, max_pages=None, deadline_ms=None, session="default") → 启动异步爬取任务,立即返回 job_id

    • 抓取语义与 web_crawl 相同;任务在后台推进,页数多时不会阻塞 MCP 请求

    • 状态逐页原子落盘到 DATA_DIR/jobs/<job_id>.json;同参数重复 start 幂等(existing=true)

    • web_crawl_status(job_id, include_results=True) → 读取状态、进度与已抓页面;进程重启后会从断点续跑

    • web_crawl_cancel(job_id) → 停止任务,已完成页面的部分结果仍可读取;终态任务幂等返回

    • web_crawl_list(limit=20) → 列出最近任务(不含正文)

  • web_watch(url, selector=None, reset=False, bypass_cache=True, ...) → 对比页面自上次抓取以来的变化

    • 每次成功抓取都把正文存一份快照到 DATA_DIR/watch/,下次调用与它对比:返回 changed、added_lines / removed_lines 与 diff 预览(最多 40 行,diff_truncated 标明是否截断)

    • first_seen=true 表示这是该 URL 的第一份快照(没有可比对象,故不给 diff);previous_hash / previous_chars 便于人工核对

    • selector 只看页面内某一块(按命中的全部节点拼接),适合盯榜单/价格/公告位;不同 selector 各存一份快照,互不覆盖

    • reset=true 丢弃旧快照重建基线;比较口径忽略行尾空白与空行,避免渲染噪声误报

    • 抓取失败(拦截/超时/登录墙)不会覆盖快照,恢复后仍能对比出期间变化;bypass_cache 默认 true,保证拿到的是最新内容

  • web_schedule(action="list", id=None, kind="watch", url=None, selector=None, interval_s=3600, webhook_url=None, session="default", ...) → 管理定时监控任务

    • create:登记"盯什么、多久一次、变化后通知谁";kind=watch 复用 web_watch(可配 selector),kind=index 复用 web_index;同 kind+url+selector 只有一条任务,重复 create 视为更新

    • interval_s 允许 60 ~ 604800(一周),默认 3600;webhook_url 只接受 http/https

    • list / get 查看任务与上次状态;pause / resume 启停(暂停后 daemon 不再执行);delete 删除调度状态(页面快照保留)

    • run 立即执行一次:首次返回 first_seen=true 且不发通知;之后仅在真实变更时推送 webhook,失败不影响任务状态

    • 实际执行由 scrape-mcp schedule daemon 驱动(见下文「定时监控」);任务状态落在 DATA_DIR/schedule/tasks.json

  • web_search(query, max_results=10, provider=None, site=None, time_range=None, engines=None, fallback=None, ...) → 搜索关键词,返回结构化结果列表

    • provider: duckduckgo(默认,取 SEARCH_PROVIDER)/ searxng(需先设 SEARCH_SEARXNG_URL)/ brave / tavily(需先设对应 key)

    • site 以 site: 运算符透传给后端(是否严格限定取决于该引擎支持程度);time_range 取 day / week / month / year;engines 为 SearXNG 引擎名单(仅对 searxng 生效)

    • 每条结果含 rank / title / url / snippet / engine / published_at;attempts 记录回退前的失败后端

    • 后端失败(限流/拦截/传输错误)时按 fallback(默认取 SEARCH_FALLBACK)自动回退到其他已配置的后端;全部不可用时行为与之前一致

    • 搜索服务按接口使用:仍遵守主机白/黑名单与全局限流,但不套目标站的 robots 抓取规则

    • bypass_cache 强制现查;proxy 与 deadline_ms 语义同 web_fetch,回退同样受总预算约束

  • web_screenshot(url, path=None, full_page=False, width=None, height=None, wait_ms=0, selector=None, hide=None, device=None, pdf=False, ...) → 浏览器渲染并截图,返回本地 PNG/PDF 路径

    • 不传 path 时存到 DATA_DIR/screenshots/;full_page=true 截整页,否则只截视口;pdf=true 导出整页 PDF(需 Chromium,忽略元素裁剪)

    • selector 截图前等待该 CSS 选择器出现,命中时按元素包围盒裁剪;未命中或元素不可见时回退整页并在 notes 说明

    • width/height 不传时取 device 预设(desktop 1280x800 / mobile 390x844,mobile 同时用移动 UA 与触屏);hide 在截图前隐藏指定元素(如固定导航/弹窗),单个失败只记 notes

    • wait_ms 额外等动画/懒加载;actions 可在截图前执行点击/输入/滚动等动作

    • 复用 session 登录态与浏览器池;proxy / deadline_ms 语义同 web_fetch

  • web_map(url, include=None, exclude=None, search=None, limit=500, homepage=True, use_search=False, ...) → 发现站点 URL(sitemap + 入口页链接)

    • 同一 URL 只出现一次,urls 每项带 source(sitemap / homepage / search),sources 给出各类计数

    • include/exclude 为正则,search 为 URL 子串过滤;limit 默认 500、硬上限 5000,超出时 truncated=true

    • 无 sitemap 的站点也能从入口页链接得到候选;use_search=true 时额外合并 site:<host> 搜索结果

  • web_index(url, include=None, exclude=None, limit=1000, check_content=False, content_diff=False, dry_run=False, reset=False, ...) → sitemap 增量索引

    • 传站点根/任意页面 URL 按 host 找 sitemap,也可直接传 sitemap 地址;include/exclude 用正则过滤 URL

    • 返回 new / changed / unchanged / removed 计数与对应 URL 列表(每类最多 50 条),first_seen 标明是否首次建立基线

    • check_content=true 对新增/变更条目回抓正文比哈希(单次最多 20 个),内容没真变的不算 changed

    • content_diff=true 在回抓正文的同时与 web_watch 共用快照,并额外返回 changes:逐 URL 的 changed / first_seen / added_lines / removed_lines / diff(截断);快照写回后单页 web_watch 可直接接着对比,dry_run=true 不写快照

    • dry_run=true 只对比不落盘;reset=true 丢弃旧基线;limit 限制本次处理条数(硬上限 5000)

  • login(url, timeout=180, session="default", headless=False, qr_selector=None, cdp_url=None) → 打开浏览器登录并持久化登录态

    • 多账号场景用不同 session 名分别登录,web_fetch 时传同一个名字即可,互不覆盖

    • headless=true:服务器无 GUI 时不弹窗口,登录页截图保存到 DATA_DIR/login/,载荷给出 screenshot 路径供查看/扫码

    • qr_selector:只截取登录二维码元素(选择器未命中时回退整页并在 notes 说明),适合手机扫码

    • cdp_url:连接远程浏览器(如另一台机器上的 Chrome)完成登录,适合本地既无 GUI 也不方便扫码的场景

    • 也可以完全绕开浏览器:用 session_import 导入浏览器扩展导出的 cookie

  • session_list() → 列出本地已保存的登录态(session + host + updated_at),不含 cookie 内容

  • session_export(url, session="default", path=None) → 导出该站 cookie({host, session, cookies});传 path 时同时写入本地文件

  • session_import(url, path, session="default", replace=False) → 从本地文件导入 cookie

    • 兼容 session_export 产物、Playwright storage_state、裸 cookie 数组,以及浏览器扩展导出的 Netscape cookies.txt

    • 默认按 (name, domain, path) 合并(同名以导入值为准),replace=true 整体替换;只读本地文件,不发网络请求

返回为 JSON 字符串(落在 content[0].text,请解析文本,勿依赖 structured_content)。 失败时带稳定 error_code 与 retryable;传输失败/5xx 会自动指数退避重试,attempts 为实际尝试次数;L2 交互未命中(如选择器不存在)只在 notes 中提示,不判失败。

错误码与诊断

error_code 是协议的一部分(只增不改),retryable 直接告诉你"过一会儿重试是否有意义":

error_code

retryable

含义 / 下一步

timeout / dns / tls / connection / unknown

✅

网络层问题,稍后重试或检查代理

http_5xx

✅

目标服务端异常,可退避重试

http_4xx

❌

404/410 等,页面本身无效

blocked

❌

被反爬拦截(Cloudflare/WAF/限流),换时段或人工确认

login_required

❌

页面确有登录入口(密码框/登录字样),先调用 login;裸 403 拒绝页没有登录入口,只报 blocked

denied / robots_disallowed

❌

黑白名单或 robots.txt 拒绝

missing_required

❌

必填字段缺失,见 required_missing;补齐 schema 或放宽 required

unsupported_content

❌

内容形态不支持当前工具(如对 PDF/Office 用 web_extract);改用 web_fetch,PDF 需装 [pdf] extra,扫描件还需 [ocr]

invalid_url / invalid_argument

❌

参数问题

失败时默认带精简 diagnostics.decisions(如 ["retry:5xx", "escalate_l2:cloudflare", "l2_ok"]); 走粘性升级时首条是 skip_l1:sticky_l2,浏览器这次没跑起来则会看到 sticky_l2_failed:fallback_l1。 传了 deadline_ms 且触发降级时,会看到 deadline:skip_retry(预算不够再等一次退避)、 deadline:skip_l2(不够启动浏览器,返回 L1 结果)或 deadline:exceeded(没来得及发起请求)。 传 debug=true 会展开更多计数:

{"diagnostics": {
  "decisions": ["escalate_l2:shell", "l2_ok"],
  "attempts": 1,
  "html_bytes": 348210,
  "root": "div",
  "selector_hits": {"article": 0, "main": 1, "div": 812},
  "content_chars": 5820,
  "l2": {"wait_for_hit": null, "clicked": null, "scroll_rounds": 0, "error": null}
}}

429/503 带 Retry-After 时返回 retry_after_s;配置 RETRY_AFTER_MAX_WAIT 后可在上限内自动补一次重试。

web_extract 示例

schema='{"fields": {
  "title":     {"selector": "h1", "required": true},
  "price":     {"selector": ".price", "regex": "¥([\\d,.]+)", "group": 1,
                "convert": "float", "required": true},
  "published": {"selector": "time", "type": "attr", "attr": "datetime", "convert": "date"},
  "main_link": {"selector": "a", "type": "attr", "attr": "href", "abs_url": true},
  "tag_count": {"selector": ".tag", "type": "count"},
  "tags":      {"selector": ".tag", "type": "list", "list_key": "text"},
  "author":    {"type": "jsonld", "path": "author.name"},
  "items":     {"selector": ".item", "type": "list", "fields": {
    "name": {"selector": ".name"},
    "url":  {"selector": "a", "type": "attr", "attr": "href", "abs_url": true}
  }}
}}'
# → {"ok":true, "data": {"title":"…","price":299.0,"published":"2026-10-20",
#    "main_link":"https://…","tag_count":3,"tags":["a","b","c"],
#    "author":"…","items":[{"name":"…","url":"https://…"}]},
#    "missing":[], "required_missing":[]}

必填字段缺失时返回 ok=false + error_code=missing_required,required_missing 里是缺失的字段名:

{"ok": false, "error_code": "missing_required", "error": "必填字段缺失: title, price",
 "retryable": false, "required_missing": ["title", "price"], "data": {"title": null, ...}}

站点 recipe

把"这个站怎么抓、取哪块、抽哪些字段、怎么翻页"写成一份 JSON 放进 RECIPES_DIR (默认 ~/.scrape_mcp/recipes),之后同一站点免参数复用。文件名即主机名, 也可用 host 字段覆盖;匹配按最长后缀(m.example.com 能命中 example.com), 单次调用的显式参数优先于 recipe。

~/.scrape_mcp/recipes/example.com.json:

{
  "host": "example.com",
  "wait_for": "main article",
  "scroll": true,
  "root_selector": "main article",
  "extract": {
    "fields": {
      "title": {"selector": "h1", "required": true},
      "author": {"type": "jsonld", "path": "author.name"},
      "published": {"selector": "time", "type": "attr", "attr": "datetime", "convert": "date"},
      "items": {
        "selector": ".item",
        "type": "list",
        "fields": {
          "name": {"selector": ".name"},
          "url": {"selector": "a", "type": "attr", "attr": "href", "abs_url": true}
        }
      }
    }
  },
  "crawl": {"next_selector": "a.next", "max_pages": 5}
}

字段

作用

生效范围

host

匹配的主机(缺省用文件名)

全部

wait_for / scroll / click

L2 渲染交互,缺省则用调用参数

web_fetch / web_extract / web_batch / web_crawl

root_selector

指定正文根,命中失败自动回落密度选根

同上

extract

web_extract 省略 schema 时的默认 schema

web_extract

crawl.next_selector / crawl.max_pages

分页翻页规则,next_selector 一旦生效即进入 mode=paginate

web_crawl

note

备注,仅供人读

—

改完 JSON 无需重启:按文件 mtime 自动重载。写坏的 JSON 会被跳过,不影响其他 recipe。

开箱示例(真实站点)

examples/recipes/ 里有八份可直接使用的 recipe,复制到 RECIPES_DIR 即可:

mkdir -p ~/.scrape_mcp/recipes
cp examples/recipes/*.json ~/.scrape_mcp/recipes/

文件

站点

演示的 recipe 能力

docs.python.org.json

Python 官方文档

root_selector 取 div.body;crawl.next_selector 用 accesskey="N" 顺序翻页;extract 取标题/版本/章节数/下一章链接

github.com.json

GitHub Trending

type=list 一次抽整张榜单(仓库名、URL、今日 star、语言、简介)

ruanyifeng.com.json

阮一峰的网络日志

正文根 #main-content + regex 提取发布日期;host 用 ruanyifeng.com 以覆盖 www. 子域

developer.mozilla.org.json

MDN 文档

正文根用 main#content(该站没有 article 标签);extract 取标题/章节数/正文链接数

fastapi.tiangolo.com.json

FastAPI 文档

Material for MkDocs 站:正文根 .md-content__inner,避开 .md-sidebar 导航

cn.vuejs.org.json

Vue 中文文档

VitePress 站:正文根 .vt-doc(旧版主题是 .vp-doc)

cnblogs.com.json

博客园首页

列表页 type=list 一次抽 20 条(标题/URL/作者/摘要/时间);翻页只有页码链接,不配 crawl

chinanews.com.cn.json

中国新闻网文章页

正文根 .left_zw,标题 required;时间/来源/作者在 BaiduSpider 隐藏块

# 按 recipe 抽取(不传 schema)
web_extract(url="https://docs.python.org/3/library/asyncio.html")
web_extract(url="https://github.com/trending")
web_extract(url="https://www.ruanyifeng.com/blog/2026/10/weekly-issue-415.html")
web_extract(url="https://developer.mozilla.org/zh-CN/docs/Web/HTTP/Guides/CORS")
web_extract(url="https://fastapi.tiangolo.com/tutorial/")
web_extract(url="https://cn.vuejs.org/guide/introduction.html")
web_extract(url="https://www.cnblogs.com/")
web_extract(url="https://www.chinanews.com.cn/gn/2026/10-21/10718387.shtml")

# 按 recipe 翻页(next_selector 生效即进入 mode=paginate)
web_crawl(url="https://docs.python.org/3/library/asyncio.html", max_pages=3)

示例选择器于 2026-10 实测通过;站点改版后失效属正常现象,改一行 selector 即可恢复。

校验与共享

写完先本地验一遍,别等运行时才发现选择器写错:

scrape-mcp recipe validate examples/recipes          # 校验目录(不传路径则校验 RECIPES_DIR)
scrape-mcp recipe validate ~/my-recipe.json          # 或单个文件
scrape-mcp recipe validate examples/recipes --strict # 警告也算失败,适合挂 CI
scrape-mcp recipe list                               # 列出已生效的 recipe:主机、文件、启用项

它会指出:JSON/编码问题(含 Windows 记事本写出的 BOM)、未知字段(带「是不是 wait_for?」式近似提示)、非法主机名、选择器语法(用运行时同一个解析器验)、extract 的 schema 错误,以及同一目录里 host 重复。退出码 0(通过)/ 1(有问题)/ 2(找不到文件),可直接用于流水线。

examples/recipes/ 就是社区配方集:加一份新配方 = 放一个 <host>.json 并用 recipe validate 验过;CI 会以 --strict 校验这八份,坏配方进不来。

分享或批量迁移时用 bundle:一个文件装多份 recipe,带 schema 版本与导出信息。

scrape-mcp recipe export --dir examples/recipes -o recipes.bundle.json  # 整个目录打包
scrape-mcp recipe export --dir examples/recipes cnblogs.com github.com  # 只导出指定 host
scrape-mcp recipe import recipes.bundle.json                      # 导入到 RECIPES_DIR
scrape-mcp recipe import recipes.bundle.json --dry-run            # 只打印计划,不写文件
scrape-mcp recipe import recipes.bundle.json --overwrite          # 覆盖同名但内容不同的本地 recipe

导出会跳过解析失败或有结构错误的文件并给出警告。导入先整体校验(schema/版本、host 重复、选择器语法、extract 规则),有任何错误就一个文件都不写;同名但内容不同的 recipe 默认记为冲突、退出码 1,避免误覆盖本地调整;内容完全相同则跳过。

单次覆盖:显式参数优先于 recipe,例如 web_fetch(url="...", wait_for=".comments")。

自定义字段类型(插件)

内置七种类型覆盖大多数页面;遇到特殊结构(meta 标签、微数据、需要跨节点计算)时,可以写个插件把自己的类型加进 web_extract,用法与内置类型完全一致。

最小插件包(本地包或发布到 PyPI 都行):

# pyproject.toml
[project]
name = "scrape-mcp-meta"
dependencies = ["mcp-scrape>=0.18"]

[project.entry-points."scrape_mcp.extractors"]
meta_content = "scrape_mcp_meta:meta_content"
# scrape_mcp_meta.py
def meta_content(ctx):
    # ctx.nodes     选择器命中的节点列表(可能为空)
    # ctx.rule      原始字段规则,可读自定义键(如 "sep")
    # ctx.root      整页解析树;ctx.base_url 页面最终地址
    if not ctx.nodes:
        return None
    return (ctx.nodes[0].attributes or {}).get("content")

装好后自动发现,直接当类型用:

{"desc": {"selector": "meta[name=description]", "type": "meta_content"}}

约定与边界:

  • 名字必须是小写标识符,不能占用内置类型名;嵌套 fields 只有内置 object / list 支持

  • 返回字符串或列表时,regex / abs_url / strip / convert / required 照常生效

  • 插件抛异常只影响该字段:记进 errors 并回落到 default,同页其他字段不受牵连;返回空值同样回落 default

  • 插件是进程内代码,装了什么就跑什么;不想引入第三方代码就别装扩展

数据集导出

把每次成功抓取累积成 JSONL,可直接喂给 pandas / DuckDB / 向量库。默认关闭,配一个路径就生效:

$env:SCRAPE_MCP_DATASET="D:\data\scrapes.jsonl"
# 或写进 .env:SCRAPE_MCP_DATASET=~/datasets/scrapes.jsonl

之后 web_fetch / web_batch / web_extract / web_crawl 每产生一条成功结果就追加一行;失败、拦截、超时都不记,免得噪声污染数据集:

{"v": 1, "ts": "2026-10-29T12:00:00+00:00", "tool": "fetch", "url": "https://example.com/a", "final_url": "https://example.com/a", "status": 200, "tier": "l1", "title": "示例", "tokens": 1234, "chars": 5678, "content": "正文……", "error_code": null}

web_extract 的结果额外带 data(字段抽取结果)。查看与转换:

scrape-mcp dataset stats                        # 条数、按工具分布、时间跨度、平均 token/字符
scrape-mcp dataset stats D:\data\scrapes.jsonl  # 也可显式指定文件
scrape-mcp dataset to-parquet                   # 生成同目录 .parquet(需 mcp-scrape[dataset])

边界:

  • 只记成功结果;同一 URL 重复抓取会重复记录,去重交给下游(url + ts 足够)

  • 写盘失败只记 warning,不影响抓取本身;导出是旁路能力

  • 文件是持续追加,长期运行请自行按天轮转(如 scrapes-2026-10-29.jsonl)

写入向量库

抓取产物可以直接进 RAG:示例脚本把 JSONL 切块后写入 Chroma 或 Qdrant,metadata 带 url / title / tool / ts / status,检索命中后能回链原文。

pip install "mcp-scrape[vector]"                  # chromadb + qdrant-client
python examples/06_vector_store.py D:\data\scrapes.jsonl --dry-run
python examples/06_vector_store.py D:\data\scrapes.jsonl --backend chroma --persist-dir .chroma
python examples/06_vector_store.py D:\data --backend qdrant --qdrant-url http://localhost:6333
  • 每块约 900 字符、相邻块重叠 120 字符;ID 由 url + ts + 块序号 稳定生成,重复导入不会产生重复点

  • --embedder auto:装了 sentence-transformers 就用 all-MiniLM-L6-v2,否则退化为无依赖的 hash 向量;后者只够验证链路,别当语义检索质量用

  • --collection / --batch / --limit 可调;--dry-run 只打印记录数、页面数、块数与向量维度,不连后端

定时监控

把 web_watch / web_index 交给本机 daemon 定时执行,变化后推送 webhook。任务只存在本地,不引入常驻云服务:

scrape-mcp schedule add --url https://example.com/news --selector "main .list" --interval 3600 \
  --webhook https://hooks.example.com/scrape
scrape-mcp schedule add --kind index --url https://example.com --interval 86400 \
  --webhook https://hooks.example.com/sitemap
scrape-mcp schedule list          # 查看 ID、状态、下次/上次执行
scrape-mcp schedule run <ID>      # 手动跑一次(首次只建基线)
scrape-mcp schedule pause <ID>    # 暂停,保留快照
scrape-mcp schedule resume <ID>
scrape-mcp schedule delete <ID>
scrape-mcp schedule daemon        # 常驻:每 30 秒扫描到期任务;Ctrl+C 退出
scrape-mcp schedule daemon --once # 只跑当前到期任务(适合系统 cron)
  • 首次执行只建立基线,不算变更、不发通知;之后 watch 比对正文哈希,index 比对 sitemap 清单的 new + changed。

  • webhook 为 JSON POST:{"event":"change","task_id","kind","url","selector","checked_at","summary"};summary 带 diff 摘要或新增/变更 URL 列表。

  • webhook 失败只体现在本次执行结果里,不改写任务状态;下一次仍会照常检测。

  • 想复用系统 cron 时,把 schedule daemon --once 放进 crontab 即可,不必常驻进程。

  • 任务状态在 DATA_DIR/schedule/tasks.json;页面快照仍在 DATA_DIR/watch/、DATA_DIR/index/,删除任务不会丢快照。

分布式执行(可选)

默认仍是单进程 + 本地 JSON。需要多实例横向扩容时,把异步 crawl 任务放进 Redis 队列:API 进程只投递任务,scrape-mcp worker 进程消费并推进,状态对所有实例共享。

pip install 'mcp-scrape[redis]'
export SCRAPE_MCP_JOB_BACKEND=redis
export SCRAPE_MCP_REDIS_URL=redis://127.0.0.1:6379/0

scrape-mcp --transport streamable-http --port 8000   # API 实例,可起多个
scrape-mcp worker                                    # 执行实例,可起多个
  • 任务状态存 <PREFIX>:job:<id>(JOB_TTL 默认保留 7 天),待执行队列为 <PREFIX>:jobs:queue;web_crawl_list 通过 <PREFIX>:jobs:index 按更新时间倒序读取。

  • 同参数重复 web_crawl_start 仍然幂等,只入队一次;status / cancel / list 读同一份共享状态。取消由 API 进程写入,worker 在下一页开始前读到后停止。

  • worker 启动时会把 running 且超过 JOB_STALE_AFTER 秒未更新的任务重新入队,进程崩溃后不必手动清理;正常推进的任务每页都会刷新时间戳,不会被误抢。

  • 未安装可选依赖或连不上 Redis 时,启动会给出明确错误;JOB_BACKEND=local 的行为与之前完全一致。

  • 注意:当前共享的只有任务状态与队列。HTTP 令牌的 QPS / 日配额计数仍在各进程内存中,多实例部署时需按实例数调整阈值。

远程浏览器池(可选)

L2 渲染可以不自启 Chromium,而是连接外部 CDP 服务(Browserless、自建 Playwright 集群,或云厂商的浏览器池):

export SCRAPE_MCP_REMOTE_BROWSER_CDP_URL=http://127.0.0.1:3000   # Browserless 示例端点
# 服务需要鉴权时,把 token 拼进 URL 查询串,如 ...?token=xxx
  • 连接成功时不再本地启动浏览器;空闲回收仍按 L2_IDLE_TIMEOUT 断开远程连接。

  • 连接失败默认回退本地渠道;REMOTE_BROWSER_FALLBACK=false 时直接报错,避免云端池不可用时悄悄换成本地出口。

  • 只改变 L2 的浏览器来源,L1 抓取与 login 的 cdp_url 参数不受影响。

  • 远程模式连接的是服务端已有浏览器,L2_CHANNEL / L2_HEADLESS 等本地启动参数不生效;启动时会校验 URL 协议。

云端控制面 broker(可选)

P2-4b 只实现控制面:把运维提供的浏览器 / 代理上游按租户分发成短租约,并做并发上限与按日配额;不转发流量、不托管浏览器进程、也不生成计费账单。默认关闭,纯本机或自建部署不受影响。

export SCRAPE_MCP_CLOUD_BROKER_ENABLED=1
export SCRAPE_MCP_CLOUD_BROWSER_UPSTREAMS=http://browser-pool-1:3000,ws://browser-pool-2:3000
export SCRAPE_MCP_CLOUD_PROXY_UPSTREAMS=socks5://proxy-us:1080|us,http://proxy-hk:8080|hk
export SCRAPE_MCP_CLOUD_LEASE_TTL=300
export SCRAPE_MCP_CLOUD_MAX_LEASES_PER_TENANT=2
export SCRAPE_MCP_CLOUD_DAILY_LEASES=100
curl -X POST -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8000/cloud/browser/lease
# {"ok":true,"lease_id":"...","kind":"browser","url":"http://browser-pool-1:3000",
#  "region":"","expires_at":"...Z","expires_in_s":300,"request_id":"..."}

curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8000/cloud/usage
# {"ok":true,"tenant":"team-a","lease_ttl_s":300,
#  "browser":{"active":1,"limit":2,"daily_used":1,"daily_limit":100,"daily_remaining":99,"upstreams":2},
#  "proxy":{...},"request_id":"..."}

curl -X POST -H "Authorization: Bearer $TOKEN" \
  "http://127.0.0.1:8000/cloud/proxy/lease?region=us"   # 只从 us 上游分配

curl -H "Authorization: Bearer $TOKEN" http://127.0.0.1:8000/cloud/leases
# {"ok":true,"tenant":"team-a","count":2,"leases":[...],"request_id":"..."}

curl -X DELETE -H "Authorization: Bearer $TOKEN" \
  "http://127.0.0.1:8000/cloud/proxy/lease/<lease_id>"  # 提前归还
  • POST /cloud/<kind>/lease 申请新租约;GET 复用当前未过期租约(没有才发放,复用不消耗日配额)。kind 为 browser 或 proxy

  • 代理租约可带 ?region=us,只从对应地区标签的上游分配,轮询游标按「类型 + 地区」独立;未知地区返回 503 cloud_unavailable,浏览器带 region 或标签格式非法返回 400 cloud_bad_request

  • 上游按轮询分配;租约到期自动回收。浏览器上游是 CDP 端点,代理上游条目可带 |region 标签

  • DELETE /cloud/<kind>/lease/<lease_id> 提前归还租约:立即空出并发名额,但不退回当日配额;不存在、已过期或不属于当前租户返回 404 cloud_lease_not_found

  • GET /cloud/leases 列出本租户全部活跃租约(浏览器 + 代理,按到期时间升序)

  • 单租户并发租约达到 CLOUD_MAX_LEASES_PER_TENANT 时返回 429 cloud_lease_limit,带 Retry-After 与 retry_after_s

  • 日租约配额(CLOUD_DAILY_LEASES,UTC 零点重置,浏览器/代理分别计数,与 HTTP_DAILY_QUOTA 独立)超限返回 429 cloud_quota_exceeded

  • /cloud/* 未启用返回 404 cloud_disabled;启用但某类上游为空返回 503 cloud_unavailable;租户身份来自 Bearer 中间件,未配置令牌的纯本机模式记为 local

  • 上游地址在启动时校验(浏览器 http/https/ws/wss,代理 http/https/socks4/socks5 且地区标签合法),非法配置直接拒绝启动

  • 租约与配额默认保存在进程内;设置 CLOUD_BROKER_REDIS_URL 后改存 Redis,多实例共享轮询游标、并发上限与日配额,写路径按「租户 + 类型」加短锁避免同时击穿;需 pip install "mcp-scrape[redis]",启动时会检查连接

快照对象存储(可选)

默认快照仍写 DATA_DIR/watch。设置 SNAPSHOT_BACKEND=s3 后,web_watch 与 web_index(content_diff) 共用同一份快照,落到 S3 兼容存储,多个实例共享基线。

pip install 'mcp-scrape[s3]'
export SCRAPE_MCP_SNAPSHOT_BACKEND=s3
export SCRAPE_MCP_S3_BUCKET=scrape-snapshots
export SCRAPE_MCP_S3_ENDPOINT_URL=http://127.0.0.1:9000   # MinIO 示例
export SCRAPE_MCP_S3_ACCESS_KEY_ID=...
export SCRAPE_MCP_S3_SECRET_ACCESS_KEY=...
export SCRAPE_MCP_S3_PATH_STYLE=true
  • 对象键为 <S3_PREFIX>/watch/<hash>.json,内容与本地快照一致(含 saved_at)。

  • 快照是旁路:上传或读取失败都按“没有快照”处理,不影响本次抓取;首次写入即建立基线。

  • 默认仍是 local;SNAPSHOT_BACKEND=s3 未设置 S3_BUCKET 或未安装 boto3 时启动直接报错。

  • 凭据留空时走 boto3 默认链(环境变量 / IAM 角色),也适合用预签名或实例角色免密部署。

数据集与任务产物归档(可选)

同样复用上面的 S3_* 配置,另有两处旁路:

  • 数据集:DATASET_BACKEND=s3 时不需要设置 DATASET 路径,每条成功结果写成一个对象 <S3_PREFIX>/dataset/<UTC 日期>/<uuid>.json,多实例并发不会互相覆盖。scrape-mcp dataset stats 直接从 S3 汇总;dataset to-parquet 需用 -o/--out 指定本地输出路径。

  • 任务产物:S3_JOB_ARTIFACTS=1 时,异步 crawl 进入终态(done / failed / cancelled)会把完整任务 JSON 归档到 <S3_PREFIX>/jobs/<job_id>.json,便于审计与离线分析。

  • 两处都是旁路:上传失败不影响抓取结果或任务状态;任一启用(或快照用 s3)都会在启动时检查 S3_BUCKET 与 boto3 依赖。

生态集成(LangChain / LlamaIndex)

适配器是薄封装:内部调用 scrape_mcp.sdk,参数、JSON 载荷与错误码和 MCP 工具完全一致。核心包不依赖这两个框架,只有显式安装可选依赖后才可用。

LangChain

pip install 'mcp-scrape[langchain]'
from scrape_mcp.integrations.langchain import build_tools

tools = build_tools()  # scrape_web_fetch / scrape_web_extract / scrape_web_search
# 交给 create_tool_calling_agent(...) 或任意支持 tool calling 的 Agent

工具返回 JSON 字符串;scrape_web_extract 的字段选择器在模型侧仍叫 schema。

LlamaIndex

pip install 'mcp-scrape[llamaindex]'
from scrape_mcp.integrations.llamaindex import build_reader

reader = build_reader()
docs = reader.load_data(["https://example.com", "https://docs.python.org/3/"])  # 同步
# 已在事件循环内时改用:docs = await reader.aload_data([...])
  • load_data() 在没有运行事件循环时内部用 asyncio.run;在协程里会提示改用 await reader.aload_data(...)。

  • 失败页直接跳过;成功的 content 成为 Document.text,url/title/status_code/content_type 进 metadata。

  • 其余参数(如 max_tokens)原样透传给 web_fetch。

  • 适配器复用进程内的同一个抓取器;脚本退出前可 await scrape_mcp.sdk.aclose() 释放浏览器。

Python SDK

不接 MCP 也能直接用同一套实现(参数、返回值、错误码与工具完全一致):

import asyncio

from scrape_mcp.sdk import (
    aclose,
    batch,
    crawl,
    crawl_cancel,
    crawl_start,
    crawl_status,
    extract,
    fetch,
    index,
    screenshot,
    search,
    site_map,
)


async def main() -> None:
    page = await fetch("https://docs.python.org/3/library/asyncio.html", max_tokens=2000)
    print(page["ok"], page["token_estimate"], page["tier"])

    # 按字段抽取;不传 schema 时套用该站 recipe
    data = await extract(
        "https://docs.python.org/3/library/asyncio.html",
        {
            "fields": {
                "title": {"selector": "h1"},
                "sections": {"selector": "section", "type": "count"},
            }
        },
    )
    print(data["data"])

    pages = await crawl(
        "https://docs.python.org/3/library/asyncio.html",
        next_selector='div.related a[accesskey="N"]',
        max_pages=3,
    )
    print(pages["mode"], pages["pages_fetched"])

    many = await batch(["https://example.com", "https://example.org"])
    print(many["ok_count"], "/", many["total"])

    hits = await search("python asyncio 文档", max_results=5)
    print(hits["count"], hits["results"][0]["url"])

    shot = await screenshot("https://example.com", full_page=True)
    print(shot["file"], shot["bytes"])

    urls = await site_map("https://docs.python.org", limit=100)
    print(urls["count"], urls["sources"])

    job = await crawl_start("https://docs.python.org", depth=1, limit=20)
    status = await crawl_status(job["job_id"])
    print(status["state"], status["progress"])

    changes = await index("https://docs.python.org", dry_run=True)
    print(changes["new"], changes["changed"], changes["removed"])

    await aclose()  # 触发过 L2 时释放浏览器子进程;不调用也能跑,只是退出时可能有清理告警


asyncio.run(main())
  • 返回的就是 MCP 工具的 JSON 载荷(已解析为 dict),失败同样带 error_code / retryable

  • SDK 放在 scrape_mcp.sdk,是为了不和内部 scrape_mcp.extract 子包重名

示例脚本

examples/ 下按场景提供 5 个可直接运行的脚本(默认参数就能跑通,需要代理时先设 SCRAPE_MCP_PROXY):

脚本

场景

常用参数

01_fetch_article.py

长文压成 token 预算内的正文

--max-tokens --out

02_extract_list.py

列表页抽结构化数据(不取整页正文)

--json --csv

03_crawl_docs.py

按"下一页"翻页并合并成一份 markdown

--selector --max-pages --out

04_pdf_to_text.py

PDF 直接读成文本(需 mcp-scrape[pdf])

--out --max-tokens

05_batch_check.py

批量巡检,输出层级/状态/token/错误码,失败即非零退出

位置参数传 URL

06_vector_store.py

dataset JSONL 切块写入 Chroma/Qdrant(需 mcp-scrape[vector])

--backend --dry-run --collection

python examples/01_fetch_article.py https://ruanyifeng.com/blog/ --max-tokens 1200
python examples/02_extract_list.py --csv trending.csv
python examples/03_crawl_docs.py --max-pages 5 --out docs.md

另有 examples/example_client.py(MCP 客户端调用示例)与 examples/recipes/*.json(站点 recipe)。

基准

三类基准,口径不同:bench.py 看缓存带来的延迟收益,bench_sites.py 看真实站点上取到的内容对不对、全不全,benchmarks/bench_core.py 看本地热路径吞吐有没有退化。

缓存与延迟

python scripts/bench.py                              # 冷/热对照
python scripts/bench.py --l2 https://news.qq.com/    # 额外看浏览器开销

"冷" = bypass_cache=true(走网络 + 精简),"热" = 普通调用(命中正文缓存)。 2026-10 实测(Windows 11 / 家宽直连,rounds=5,max_tokens=4000):

页面

层级

冷 p50

热 p50

提升

token

example.com

L2

495 ms

1.4 ms

~360×

330

docs.python.org(asyncio-task)

L1

681 ms

1.0 ms

~650×

3588

L2 浏览器:进程内首次调用 ≈3.5 s(含启动),复用 p50 ≈1.8 s。 网络与目标站状态会明显影响绝对值,重点看同一台机器上的冷热对比。

真实站点质量

python scripts/bench_sites.py                          # 全部站点
python scripts/bench_sites.py --nightly                # 夜间回归子集
python scripts/bench_sites.py --category docs,list     # 按分类过滤
python scripts/bench_sites.py --json out.json --md out.md
python scripts/bench_sites.py --baseline out.json      # 与上次结果对比
python scripts/bench_sites.py --nightly --baseline benchmarks/baseline.json  # 与仓库基线对比
pip install trafilatura                                # 可选:对照第三方正文抽取器
python scripts/bench_sites.py --reference trafilatura

站点集在 benchmarks/sites.json,按文档 / 博客 / 资讯 / 列表 / 前端渲染 / 登录墙与反爬 / PDF 分层,每站可配置:

  • expect:ok 表示应取到正文;login_required / robots_disallowed 等错误码表示该站应被如实拦截,而不是伪装成功;也可以写成列表(如 ["ok", "login_required"]),表示不同出口下结果不同、任一都算符合预期

  • min_chars:正文字符数下限

  • checks:正文中必须出现的稳定片段,用于衡量完整度(只给结构稳定的页面配置,首页类动态内容不配)

输出包含通过率、取正文率、拦截符合率、完整度、token、p50/p95 延迟与分层分布,可写 JSON / Markdown 留档。 --baseline 会按站点对比上次结果,并把正文字符下降超过 20% 的站点标为退化。 同一份基线还用于耗时门禁:某站耗时超过 max(基线 × 3, 基线 + 3000ms) 即判为退化、退出码非零,夜间工作流据此开 issue;两个阈值可用 --latency-factor / --latency-floor-ms 调整。

仓库里保留一份夜间子集基线 benchmarks/baseline.json,2026-10-28 随扩容重录(14/14 通过,p50 504 ms / p95 1642 ms)。它只覆盖夜间子集,用于看趋势,不代表全量站点通过率;每月挑一次夜间结果更新:下载当次 bench-sites.json,把 sites_file 改为相对路径、补上 source 字段后替换该文件。

本机实测(2026-10-28,37 站):通过 33/37,完整度 33/33,拦截符合 4/4,延迟 p50 1705 ms / p95 9011 ms。 未通过的 4 项里,3 项是本机网络不可达(Hacker News / V2EX / arXiv 超时),1 项被站点拦截(Bilibili 返回 login_required); 这些站点在 GitHub 网络的夜间任务里会重新跑,结果以 site-quality-<run_id> 工件为准。 CI 全量(2026-10-28):通过 37/37,取正文 32/33,拦截符合 4/4,完整度 33/33,延迟 p50 1456 ms / p95 4953 ms。 其中澎湃首页在 GitHub Actions 出口被站点 WAF 恒定 403(三次全量运行都是 status=403、blocked=waf、0 字),而同站的文章页 thepaper-article 正常取到 2827 字,故首页按「如实上报拦截」计入,期望值写为 ["ok", "blocked", "login_required"]:两种出口都不算退化,正文质量由文章页覆盖。 CI 夜间子集(2026-10-28,14 站):通过 14/14,取正文 13/13,拦截符合 1/1,完整度 16/16,p50 504 ms / p95 1642 ms。

站点

分类

层级

结果

字符

token

完整度

耗时

py-asyncio

docs

l1

通过

31049

7222

2/2

2001 ms

mdn-fetch

docs

l1

通过

2639

1110

2/2

2975 ms

fastapi-first-steps

docs

l1

通过

8777

4124

2/2

1705 ms

vue-intro

docs

l1

通过

4833

3101

2/2

2479 ms

py-json

docs

l1

通过

27368

6903

2/2

256 ms

django-tutorial

docs

l1

通过

8026

4243

2/2

2318 ms

rust-ownership

docs

l1

通过

24033

5740

2/2

1717 ms

ruanyifeng-blog

blog

l1

通过

908

543

2/2

2432 ms

cnblogs-home

blog

l1

通过

5135

3976

-

218 ms

sspai-home

blog

l1

通过

1613

1285

2/2

511 ms

solidot-home

blog

l1

通过

8862

7166

2/2

553 ms

cnblogs-article

blog

l1

通过

5883

5267

-

75 ms

meituan-tech

blog

l1

通过

10561

7472

1/1

1611 ms

chinanews-home

news

l1

通过

935

207

-

149 ms

tencent-news-home

news

l2

通过

2022

1378

-

4893 ms

thepaper-home

news

l2

通过

2025

1649

-

11940 ms

36kr-home

news

l2

通过

1119

1060

-

2437 ms

thepaper-article

news

l2

通过

2829

3090

1/1

1520 ms

netease-home

news

l1

通过

10485

7313

1/1

406 ms

sina-home

news

l1

通过

6360

7549

1/1

330 ms

github-trending

list

l1

通过

4913

1787

-

3682 ms

hacker-news

list

l1

timeout(本机网络)

0

0

-

45005 ms

v2ex-home

list

l1

timeout(本机网络)

0

0

-

45020 ms

stackoverflow-questions

list

l2

按预期拦截

0

0

-

11110 ms

douban-home

list

l1

通过

1354

1028

1/1

330 ms

react-learn

spa

l1

通过

13986

3514

2/2

597 ms

apple-cn-home

spa

l2

通过

198

90

2/2

1564 ms

svelte-dev

spa

l2

通过

2320

620

1/1

5797 ms

tailwind-home

spa

l1

通过

6240

1778

1/1

3773 ms

zhihu-question

anti_bot

-

按预期拦截

0

0

-

107 ms

xiaohongshu-explore

anti_bot

-

按预期拦截

0

0

-

76 ms

weibo-home

anti_bot

-

按预期拦截

0

0

-

106 ms

bilibili-home

anti_bot

l2

login_required(站点拦截)

0

0

-

2919 ms

taobao-home

anti_bot

l2

通过

5234

7563

-

2472 ms

arxiv-attention

pdf

l1

timeout(本机网络)

0

0

-

45007 ms

w3c-dummy-pdf

pdf

l1

通过

14

3

1/1

1333 ms

pdf-orimi

pdf

l1

通过

391

103

1/1

9011 ms

--reference trafilatura 的对照只统计纯正文;本工具输出保留标题、列表与链接结构,token 数不是等价口径,只看量级即可。

核心路径吞吐(CI 门禁)

benchmarks/bench_core.py 用固定种子的合成页面(约 370 KB HTML)测不依赖网络的本地热路径,CI 每次 push / PR 都跑一遍,低于 benchmarks/core_baseline.json 的 ops/sec 下限即失败:

python benchmarks/bench_core.py            # 打印表格
python benchmarks/bench_core.py --check    # 回归门禁(CI 用法)
python benchmarks/bench_core.py --json out.json
python benchmarks/bench_core.py --update-baseline   # 全量测量后重建下限(需人工确认)

指标

路径

参考机实测

compact_html

HTML → Markdown(解析 + 密度选正文 + 渲染)

~18 ops/s(~56 ms/次)

chunk_markdown

Markdown → RAG 分块(标题路径 + 窗口 + token 计数)

~18 ops/s(~56 ms/次)

build_record

工具结果 → 数据集记录(纯内存)

~44 万 ops/s

record_result

记录 → JSONL 追加写(含锁与磁盘)

~600 ops/s(~1.7 ms/次)

recipe_validate

示例配方校验(8 份)

~115 ops/s

参考机为 Windows 11 / Python 3.13;下限取实测的 35%(build_record 这种微秒级纯内存操作取 1/10,抖动更大),只拦数量级退化(例如误引入 O(n²)),不做跨机器绝对耗时比较,因此 CI 机器慢一倍也不会误报。

Docker

# 预构建镜像(GHCR:main 推 latest,v* 标签推语义化版本)
docker pull ghcr.io/guanyuyan/scrape-mcp:latest

# 或者在本地构建;带 L2 真浏览器(Cloudflare/验证码/SPA)时把 INSTALL_BROWSER 设为 1
docker build -t scrape-mcp .
docker build --build-arg INSTALL_BROWSER=1 -t scrape-mcp:l2 .

docker run -d --name scrape-mcp -p 127.0.0.1:8002:8002 \
  -e SCRAPE_MCP_HTTP_TOKEN=<令牌> \
  -v scrape_data:/data \
  scrape-mcp
curl http://127.0.0.1:8002/healthz     # {"ok":true,"version":"…"}

也可以直接 docker compose up -d --build(在 .env 里设置 SCRAPE_MCP_HTTP_TOKEN)。

  • 镜像多阶段构建,运行阶段以非 root(uid 10001)启动;/data 卷保存登录态与两层缓存

  • 容器绑定 0.0.0.0,没有令牌会拒绝启动;/healthz 免鉴权,供 HEALTHCHECK 使用

  • 反向代理、容量规划、升级与威胁模型见 DEPLOYMENT.md

  • 多实例 + Redis 共享云端租约的参考编排见 deploy/cloud(nginx + 两个实例 + Redis),完整步骤见 DEPLOYMENT.md

目录结构

benchmarks/
├── sites.json         # 真实站点质量基准站点集(分层、期望结果、完整性检查项)
├── baseline.json      # 夜间子集基线(趋势参照,每月更新)
├── bench_core.py      # 本地热路径吞吐基准(固定语料,CI 门禁)
└── core_baseline.json # 核心路径 ops/sec 下限(参考机实测 × 35%,微秒级指标取 1/10)
src/scrape_mcp
├── server.py          # MCP 入口 + 工具(web_fetch/web_extract/web_batch/web_crawl/web_map/web_crawl_*/web_schedule/login/session_*)
├── http_auth.py       # HTTP Bearer 鉴权中间件 + 非回环绑定校验
├── cli.py             # 子命令:init(客户端配置)/ schema(导出工具定义)/ schedule(定时任务 daemon)
├── sdk.py             # Python SDK 薄封装(fetch/extract/crawl/crawl_*/site_map/batch/watch/aclose)
├── integrations/      # LangChain / LlamaIndex 适配(可选依赖)
├── config.py          # 环境变量配置
├── metrics.py         # Prometheus 文本格式指标(/metrics 端点)
├── tokenizer.py       # tiktoken 计数 + 预算截断
├── core/
│   ├── http_client.py # L1 curl_cffi + 磁盘缓存
│   ├── fetcher.py     # 分级调度(L1→L2),对工具层返回 FetchOutcome
│   ├── detector.py    # 拦截/骨架页判定
│   ├── browser.py     # L2 Playwright + stealth
│   ├── session.py     # 登录态持久化(多会话 + cookie 导入导出)
│   ├── recipes.py     # 站点 recipe(按 host 最长后缀匹配、mtime 重载)
│   ├── recipe_bundle.py # recipe 导出/导入 bundle(schema 版本、整体校验)
│   ├── cache.py       # HTTP 响应缓存 + 正文/抽取结果缓存
│   ├── stats.py       # 进程内计数(/stats 端点数据源)
│   ├── jobs.py        # 异步爬取任务(状态落盘、进度、恢复与取消)
│   ├── jobstore_redis.py  # 可选 Redis 后端(共享任务状态与队列,供 worker 横向扩容)
│   ├── watch.py       # 变更检测(正文快照 + diff 摘要)
│   ├── objectstore.py  # 快照存储后端(本地 / S3 兼容)
│   ├── schedule.py    # 定时任务(状态落盘、到期计算、启停)
│   ├── errors.py      # 统一错误码
│   └── hostlimit.py   # 同域并发闸门
└── extract/
    ├── compact.py     # DOM 剪枝 → 极省 token 正文
    ├── schema.py      # 结构化抽取(按字段 schema 出 JSON)
    ├── links.py       # 链接发现(web_crawl 用)
    ├── pdf.py         # PDF → 文本(可选依赖 pypdf)
    ├── documents.py   # DOCX/XLSX/PPTX → 文本(标准库,无额外依赖)
    ├── ocr.py         # 扫描 PDF 的 OCR(可选依赖 pymupdf + pytesseract)
    └── render.py      # markdown → 纯文本 / 分块

开发

pip install -e ".[dev]"
python -m pytest

Available Tools

18 tools
loginA

打开浏览器登录,并把该站登录态持久化到磁盘。

用于 web_fetch 报告 login_required=true 的站点:默认弹出可见窗口,请在里面完成登录; 检测到登录成功或超时后,登录态(cookie 等)会被保存,之后该站的 web_fetch 会自动带上。

Args: url: 目标站点任一页面 URL(按域名区分登录态)。 timeout: 等待手动登录完成的秒数,超时也会保存当前状态便于下次继续。 session: 会话名(默认 default)。需要多账号/多身份时分别用不同名字登录, web_fetch 时传同一个 session 即可,互不覆盖。 headless: 无 GUI 服务器上设为 true:不弹窗口,把登录页截图保存到 DATA_DIR/login/,看截图或扫码完成登录;也可改用 session_import 导入 cookie。 qr_selector: 登录页上二维码元素的 CSS 选择器;命中时只截二维码,便于手机扫码。 cdp_url: 远程浏览器 CDP 地址(http/https/ws/wss);给出时连到该浏览器登录, 适合本地无 GUI 但另一台机器能开浏览器的场景。

NameRequiredDescriptionDefault
urlYes
cdp_urlNo
sessionNodefault
timeoutNo
headlessNo
qr_selectorNo

Output Schema

NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description bears the full behavioral burden, and it largely does: default visible window, manual-login wait with timeout, state saved even on timeout for later resumption, headless screenshots written to DATA_DIR/login/, and CDP remote-browser support. What it does not say is what happens to previously stored state for the same session (overwrite vs. merge) or what the return/output looks like — though an output schema exists, so the latter is partly excused.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose and trigger are front-loaded before a structured Args block, and each parameter entry adds a real operational fact rather than restating the type. It runs long relative to a typical definition, but there is little filler; the qr_selector and cdp_url entries could be tightened slightly.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a multi-modal browser-automation tool with 6 parameters and zero schema coverage, the description covers the visible-window path, the headless path, the QR path, the remote-CDP path, the multi-session path, and the timeout-recovery path. Return values are covered by the output schema, so nothing an agent needs in order to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must carry all parameter meaning, and it does: url scopes state by domain, timeout is the manual-login wait window, session is the multi-identity partition key with the crucial note that web_fetch must reuse the same name, headless explains the no-GUI fallback including the screenshot directory, qr_selector is a CSS selector that narrows the capture to the QR code, and cdp_url accepts http/https/ws/wss remote endpoints.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The first sentence states a concrete verb and effect: open a browser, log in, and persist the site's login state to disk. It also names the triggering condition (web_fetch returning login_required=true), which separates it from sibling session_import and from web_fetch itself.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit when-to-use is given ('sites where web_fetch reports login_required=true') along with alternatives: session_import for cookie import, and cdp_url for the case where the local machine has no GUI but another machine has a browser. Multi-account scenarios are routed to separate session names with a note that web_fetch must pass the same session.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

session_exportA

导出某站登录态里的 cookie,便于备份或在另一台机器上导入。

Args: url: 目标站点任一页面 URL(按域名匹配登录态)。 session: 会话名(默认 default)。 path: 可选本地文件路径;给出时把导出 JSON 写入该文件(目录需已存在), 不传则直接在返回里给出 cookies。

NameRequiredDescriptionDefault
urlYes
pathNo
sessionNodefault

Output Schema

NameRequiredDescription
resultYes

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must carry the burden — and it does disclose the key side effect: if path is given the JSON is written to a local file and the directory must already exist, otherwise cookies come back inline. It omits anything about overwriting an existing file, permission/auth requirements, or the sensitivity of the exported cookies.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose is front-loaded in one sentence and the Args block is terse and useful; every line earns its place. Minor redundancy in restating the default value and the two-line wrap on path.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values needn't be spelled out, and the description's conditional (file vs inline) behavior covers the meaningful return branch. For a tool that writes a credential-bearing file with no annotations, an explicit note on overwrite behavior and the sensitivity of the exported cookies would close the remaining gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, but the description documents all three params with real meaning beyond the schema: url is any page URL of the target site and domain-matching is how the session is resolved, session names the session with a stated default, and path is optional with a directory-must-exist precondition and a defined alternative behavior when omitted.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (导出/export) and resource (cookies from a site's login session), with the scope rule '按域名匹配登录态'. It implicitly distinguishes itself from session_import by framing the output as something to back up or import elsewhere, but it never names the sibling or states the boundary explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The opening sentence gives clear use contexts: backup, or moving a session to another machine. It does not state any exclusions or explicitly point to session_import / session_list as the alternative path, so the when-not guidance is absent.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

session_importA

从本地文件导入 cookie 到某站登录态(仅本地文件,不发起网络请求)。

文件内容支持:session_export 导出的对象(含 cookies)、Playwright storage_state、直接的 cookie 数组,或浏览器扩展导出的 Netscape cookies.txt。

Args: url: 目标站点任一页面 URL(按域名匹配登录态)。 path: 本地文件路径(容器部署时用 docker cp 放进卷,再用容器内路径)。 session: 会话名(默认 default);与 web_fetch 使用的会话名保持一致。 replace: True 时用文件里的 cookie 整体替换该会话;默认 False 为按 (name, domain, path)合并,同名以导入值为准。

NameRequiredDescriptionDefault
urlYes
pathYes
replaceNo
sessionNodefault

Output Schema

NameRequiredDescription
resultYes

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations at all, the description carries the full burden and largely meets it: it discloses that no network request is made, and critically explains the destructive semantics of replace=True (wholesale replacement of the session's cookies) versus the default merge by (name, domain, path) where imported values win on conflict. Missing only edge behaviors such as error handling for malformed files or limits on file size.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the core action and its key constraint, followed by supported file formats and then per-argument notes. Every section is useful, though the Args block restates types already present in the schema and could be tightened slightly.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 4-parameter tool with zero schema documentation and no annotations, the description covers file formats, container path handling, session naming, and merge-versus-replace semantics. An output schema exists, so unexplained return values are not a gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must supply all parameter meaning, and it does: url is domain-matched to a page, path includes a practical docker cp note for container deployments, session is named and tied to web_fetch consistency, and replace documents both its default and its exact merge semantics. This is substantially more than the schema conveys.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening sentence names a specific verb (导入/import) and resource (cookie → 站点登录态) and immediately bounds the scope with '仅本地文件,不发起网络请求'. This clearly separates it from sibling tools like login (interactive auth) and session_export (the inverse direction).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The definition establishes context well: it is the counterpart to session_export, it must share the session name used by web_fetch, and it explicitly takes a local file rather than performing network auth. It stops short of an explicit 'use login instead when...' routing statement, so a 4 rather than a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

session_listA

列出本地已保存的登录态(会话名 + 主机 + 更新时间),不含 cookie 内容。

NameRequiredDescriptionDefault

No parameters

Output Schema

NameRequiredDescription
resultYes

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It usefully discloses that only locally saved sessions are returned and that cookie contents are excluded, which signals a safe metadata-only read. However, it does not cover permissions, what happens with no saved sessions, or any behavior beyond the data scope.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with the resource and returned fields in a parenthetical, plus one scoping exclusion. No wasted text for a simple tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be explained further, and the tool has no parameters to document. The description is complete enough for correct invocation; only the absence of explicit sibling routing keeps it from a 5.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, so the baseline is 4. The description adds the useful note that the output is limited to metadata, but there are no parameters whose semantics need explaining.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (列出/list) and resource (本地已保存的登录态), and enumerates the returned fields (session name, host, update time). The explicit '不含 cookie 内容' implicitly separates it from session_export, though neither sibling is named directly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the zero-parameter read-only nature of listing sessions, but there is no explicit 'use this when...' or exclusion guidance pointing agents to session_export/import for cookie data. The agent must infer when to prefer this over the export/import siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_batchA

并发抓取一批 URL,各自走完整分级链路,返回逐个结果。

Args: urls: 目标 URL 列表(最多 batch_max_urls 个,默认 20)。 max_tokens: 每个 URL 的正文 token 预算上限。 link_policy: 同 web_fetch。 format: 同 web_fetch(markdown / text / json / chunks)。 chunk_size: format=chunks 时的窗口字符数;对所有 URL 生效。 chunk_overlap: format=chunks 时相邻块的重叠字符数;对所有 URL 生效。 headers: 所有 URL 共用的自定义请求头;传入时不走共享缓存。 cookies: 所有 URL 共用的自定义 cookie;传入时不走共享缓存。 timeout: 单次请求超时秒数。 deadline_ms: 整批共用的总预算(毫秒),不传表示不限。预算耗尽后还没开始的 子任务直接返回 timeout(diagnostics 为 deadline:exceeded),不再占用网络与浏览器。 bypass_cache: 跳过并绕过响应缓存。 debug: 每条结果附带 diagnostics(决策轨迹与计数,不含页面内容)。 session: 登录态会话名(默认 default);对该批所有 URL 生效。 proxy: 该批所有 URL 共用的代理;不传则按代理池分配,未配池时沿用全局配置。 显式传入代理时该批不读也不写共享缓存。 region: 该批所有 URL 共用的地区定向(如 us);只在走代理池时生效, 没有该地区出口时直接返回 invalid_argument。 ctx: MCP 上下文,由框架注入,用于逐条上报进度(客户端请求进度时才发通知)。

NameRequiredDescriptionDefault
urlsYes
debugNo
proxyNo
formatNomarkdown
regionNo
cookiesNo
headersNo
sessionNodefault
timeoutNo
chunk_sizeNo
max_tokensNo
deadline_msNo
link_policyNointernal
bypass_cacheNo
chunk_overlapNo

Output Schema

NameRequiredDescription
resultYes

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations at all, the description carries the full burden and does substantial work: it explains that a request passing headers, cookies, or an explicit proxy bypasses the shared cache in both directions, that region only applies when a proxy pool is used and otherwise fails invalid_argument, and that the shared deadline causes not-yet-started subtasks to return timeout with diagnostics deadline:exceeded without touching network or browser. Remaining gaps are concurrency limits and the exact meaning of batch_max_urls.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose sentence is front-loaded and the Args block is dense but earns its space given 15 undocumented schema parameters. Some phrasing is more verbose than needed (the multi-line deadline and proxy explanations), but nothing is filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 15-parameter batch tool with no annotations and an output schema that already covers return values, the description supplies most of what an agent needs: per-URL vs whole-batch scope is clear for max_tokens, deadline_ms, headers, cookies, proxy, region, and session. The unresolved pieces are the batch_max_urls value and any notion of concurrency or ordering of results.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must document all 15 parameters, and it largely does — units are given for chunk_size/chunk_overlap (characters), timeout (seconds), deadline_ms (milliseconds), and max_tokens is scoped per URL rather than per batch, which prevents a real mistake. The weak spot is link_policy, documented only as '同 web_fetch' with no values or meaning, and format's enumeration is listed but not explained.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening line states a specific verb and resource — concurrent fetching of a batch of URLs, each through the full tiered pipeline, with per-URL results returned. This distinguishes it from the singular web_fetch, though the differentiation is carried by the word 'batch' rather than an explicit contrast statement.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied: batching multiple URLs is obviously the point, and several parameters are defined by reference to web_fetch ('同 web_fetch'). There is no explicit statement of when to prefer this over N separate web_fetch calls, nor any when-not guidance or sibling routing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_crawlA

从起始页发现站内链接并批量抓取,或按"下一页"顺序翻页,适合把一个栏目的前 N 篇读进来。

每页都走完整的分级抓取与合规链路(robots/黑白名单/限流/同域并发), 单页失败只影响该页,不中断整次爬取;静态资源链接(图片/压缩包等)自动跳过。

两种模式:

  • 链接模式(默认):按 depth 分层发现站内链接,limit 控制总页数。

  • 分页模式:传 next_selector(或站点 recipe 配置),顺序跟进"下一页",max_pages 控制页数。

Args: url: 起始页 URL。 depth: 链接模式的跳转层数。0=只抓起始页;1=再抓起始页上的链接(默认);最大 3。 limit: 链接模式最多抓取的页面总数(含起始页),受配置 crawl_max_pages 上限约束。 same_host: 只跟进同主机链接(默认 True)。 max_tokens: 每页正文 token 预算上限。 link_policy: 同 web_fetch。 next_selector: 分页模式:指向"下一页"的 CSS 选择器;有值时忽略 depth。 max_pages: 分页模式最多翻多少页(默认 5),受 crawl_max_pages 上限约束。 headers: 所有页面共用的自定义请求头;传入时不走共享缓存。 cookies: 所有页面共用的自定义 cookie;传入时不走共享缓存。 timeout: 单次请求超时秒数。 deadline_ms: 整次爬取的总预算(毫秒),不传表示不限。预算耗尽后不再抓新页, 已抓到的页面照常返回;排队中的子任务直接返回 timeout。 bypass_cache: 跳过并绕过响应缓存。 session: 登录态会话名(默认 default);对本次所有页面生效。 proxy: 本次爬取所有页面共用的代理;不传则按代理池分配,未配池时沿用全局配置。 显式传入代理时不走共享缓存。 region: 本次爬取所有页面共用的地区定向(如 us);只在走代理池时生效, 没有该地区出口时直接返回 invalid_argument。 ctx: MCP 上下文,由框架注入,用于逐页上报进度(客户端请求进度时才发通知)。

NameRequiredDescriptionDefault
urlYes
depthNo
limitNo
proxyNo
regionNo
cookiesNo
headersNo
sessionNodefault
timeoutNo
max_pagesNo
same_hostNo
max_tokensNo
deadline_msNo
link_policyNointernal
bypass_cacheNo
next_selectorNo

Output Schema

NameRequiredDescription
resultYes

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are supplied, so the description carries the full burden and does so: it discloses the compliance pipeline (robots/allow-deny lists/rate limiting/same-host concurrency), per-page failure isolation ("单页失败只影响该页"), automatic skipping of static assets, and deadline semantics (queued subtasks return timeout while fetched pages are still returned). It also notes cache side effects (headers/cookies/explicit proxy bypass the shared cache) and the region failure mode returning invalid_argument. This is exactly the beyond-schema behavior an agent needs before a multi-page spend.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded summary sentence, then a short behavior paragraph, a two-bullet mode block that resolves the depth-vs-next_selector ambiguity, and a flat arg list. Nothing is repetitive against the schema and the structure matches how an agent would decide (mode first, then knobs). The single framework-injected ctx line is the only non-essential text.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 16-parameter crawling tool this is complete: entry conditions, both operating modes, per-page fault tolerance, budget/deadline behavior, cache and session interactions, and error cases are all covered. An output schema exists, so the omission of return-value details is appropriate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% across 16 parameters, so the description must compensate, and it does: every argument gets a one-line meaning plus non-obvious constraints (depth default 1 / max 3, limit bounded by crawl_max_pages, same_host default True, max_tokens per-page budget, deadline_ms as a total budget with partial-return semantics, session applying to all pages, region only effective on the proxy pool). Only link_policy defers to a sibling ("同 web_fetch") without defining the values.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a concrete verb + resource: discover in-site links from a start page and batch-fetch them, or follow "next page" sequentially; the added "适合把一个栏目的前 N 篇读进来" gives a recognizable use shape. The two named modes (链接模式 / 分页模式) further pin down what the tool does. It stops short of naming which sibling it replaces (web_fetch for one page, web_batch for a known URL list).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states the mode-selection rule: pass next_selector (or rely on a site recipe) to enter pagination mode, and "有值时忽略 depth". Scope limits are given for both modes (depth max 3, limit capped by crawl_max_pages, max_pages default 5). Missing is an explicit when-not clause pointing at web_fetch/web_batch for single-page or fixed-list cases.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_crawl_cancelA

取消一个未完成的异步爬取任务;终态任务原样返回(幂等)。

Args: job_id: web_crawl_start 返回的任务 ID。

NameRequiredDescriptionDefault
job_idYes

Output Schema

NameRequiredDescription
resultYes

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden; it does disclose two valuable traits — idempotency and that terminal jobs are returned unchanged — which is real behavioral context. However, it says nothing about whether cancellation discards partial results, whether it is reversible, or any auth/permission requirements for a mutating operation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The core behavior and idempotency guarantee are front-loaded in one compact sentence, with the parameter note following. The Args block is slightly redundant in structure but earns its place by adding the provenance of job_id.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be explained, and the description covers the key semantic (idempotent cancel). The remaining gap — consequences for partial crawl artifacts and permission needs — is modest for a single-parameter cancel tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% and the schema only labels the field 'Job Id', so the description must compensate — and it does, stating that job_id is the task ID returned by web_crawl_start, which tells the agent where to obtain the value. Format details (string type) are already in the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (取消/cancel) and resource (未完成的异步爬取任务, an unfinished async crawl job), which is clearly distinct from siblings like web_crawl_start and web_crawl_status. It also discloses the terminal-state behavior, but does not explicitly name the sibling tools it differs from.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by 'cancel an unfinished async crawl task' and the idempotency note tells the agent that calling it on a finished job is harmless, but there is no explicit when-to-use / when-not-to-use guidance or reference to web_crawl_status for checking state first.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_crawl_listA

列出最近的异步爬取任务(按更新时间倒序),不返回页面正文。

Args: limit: 最多返回多少个任务(默认 20,上限 100)。

NameRequiredDescriptionDefault
limitNo

Output Schema

NameRequiredDescription
resultYes

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden. It usefully discloses sort order and that bodies are omitted, but says nothing about scope (global vs per-session), auth requirements, or pagination beyond the limit cap.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One front-loaded sentence plus a single parameter note; nothing is redundant or padded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return shape need not be described, and the ordering and body-exclusion notes cover the main behavioral questions for a one-parameter list tool. Only the scope of the listing remains ambiguous.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% and the schema only records default 20, so the description is the sole source of the upper bound of 100 and the meaning of limit. That is a genuine addition beyond the schema, though it does not say what happens if limit exceeds 100.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: list recent async crawl tasks, ordered by update time descending, explicitly excluding page bodies. That distinguishes it from web_crawl_status (single task) and from content-returning siblings, though it never names those siblings directly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase '不返回页面正文' implies that an agent needing page content should use web_fetch/web_extract, and '异步爬取任务' implies the crawl_start/status/cancel family. But no alternative is named and no explicit when-to-use vs web_crawl_status or session_list is given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_crawl_startA

启动一个异步爬取任务,立即返回 job_id;页数多时不再阻塞 MCP 请求。

与 web_crawl 的抓取语义一致,但任务在后台推进、状态落盘到 DATA_DIR/jobs/:

  • 同一组参数重复 start 是幂等的,会返回已有 job_id(existing=true);

  • 进程重启后调用 web_crawl_status 会自动续跑未完成任务;

  • 用 web_crawl_cancel 可随时停止,已完成页面的部分结果仍可读取;

  • 任务状态逐页原子写入,不会因为中途退出而留下半个 JSON。

Args: url: 起始页 URL。 depth: 链接模式的跳转层数(0~3,默认 1)。 limit: 链接模式最多抓取的页面总数(含起始页),受 crawl_max_pages 上限约束。 same_host: 只跟进同主机链接(默认 True)。 max_tokens: 每页正文 token 预算上限。 link_policy: 同 web_fetch。 next_selector: 分页模式:指向"下一页"的 CSS 选择器;有值时忽略 depth。 max_pages: 分页模式最多翻多少页(默认 5),受 crawl_max_pages 上限约束。 deadline_ms: 任务的总预算(毫秒),不传表示不限;超时后不再抓新页。 session: 登录态会话名(默认 default)。 proxy: 本次任务所有页面共用的代理;不传则按代理池分配,未配池时沿用全局配置。

NameRequiredDescriptionDefault
urlYes
depthNo
limitNo
proxyNo
sessionNodefault
max_pagesNo
same_hostNo
max_tokensNo
deadline_msNo
link_policyNointernal
next_selectorNo

Output Schema

NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations exist, so the description carries the full burden and does so: idempotent start returning existing=true, state persisted under DATA_DIR/jobs/, automatic resumption on restart, cancellation preserving partial page results, and per-page atomic writes preventing half-written JSON. These are exactly the non-obvious traits an agent needs before launching a background job.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Behavioral contract is front-loaded before the Args block, and every line of the Args block is load-bearing given zero schema coverage. Slightly long overall, and the single line delegating link_policy to web_fetch is the one entry that doesn't earn its place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an 11-parameter async job tool with no annotations, the definition covers lifecycle (start/status/cancel), failure/restart semantics, resource limits, and the returned job_id; an output schema exists so return values need no further explanation. Nothing an agent needs to invoke it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the description compensates for all 11 params: depth range 0~3 with default, limit constrained by crawl_max_pages, next_selector switching to pagination mode and overriding depth, deadline_ms as a total budget that stops new pages, and proxy pool fallback behavior. Only link_policy is under-specified ("同 web_fetch").

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a specific verb+resource+behavior: starts an asynchronous crawl job and immediately returns a job_id. It explicitly contrasts with the synchronous sibling web_crawl ("与 web_crawl 的抓取语义一致,但任务在后台推进"), so an agent can pick between them without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

States the selecting condition ("页数多时不再阻塞 MCP 请求") and names the lifecycle siblings web_crawl_status and web_crawl_cancel with their roles (resume after restart, stop at any time). It never states the inverse case (small/single-page work → web_crawl or web_fetch), so the routing guidance is clear but not fully closed.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_crawl_statusA

读取异步爬取任务的状态与部分结果;未完成的任务会自动续跑。

Args: job_id: web_crawl_start 返回的任务 ID。 include_results: False 时只返回进度与计数,不返回已抓页面正文(适合轮询)。

NameRequiredDescriptionDefault
job_idYes
include_resultsNo

Output Schema

NameRequiredDescription
resultYes

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden, and it discloses a non-obvious behavioral trait: unfinished jobs are automatically resumed, meaning this call can advance work rather than being a pure passive read. It also explains the payload trade-off of include_results. Auth requirements, error/terminal states, and result retention are not covered, which keeps it from a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One lead sentence states the purpose and the auto-resume behavior, followed by a compact Args block covering both parameters. Every sentence adds information; nothing is redundant or buried.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be described. Purpose, both parameters, and the auto-resume side effect are all covered for a two-parameter tool. Minor gaps remain around job lifecycle states and whether partial results persist after completion.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate, and it does: job_id is defined by its origin (returned by web_crawl_start), and include_results is explained in terms of what is and is not returned. Both parameters are covered with meaning beyond their names.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource: read the status and partial results of an asynchronous crawl job. The clause '未完成的任务会自动续跑' further distinguishes it from static status reads, and naming web_crawl_start as the source of job_id ties it to a sibling. It does not explicitly contrast with web_crawl_list or web_crawl_cancel, so a 5 is not warranted.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives clear usage context: job_id comes from web_crawl_start, and include_results=False is recommended for polling because page bodies are then omitted. No explicit exclusions or guidance on when to prefer web_crawl_list or web_crawl_cancel, so it falls short of the when/when-not/alternatives bar.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_extractA

抓取网页并按字段 schema 抽取结构化 JSON(字段级数据,可入库)。

适合需要"数据而非整页正文"的场景:给 URL 和字段规则,直接返回字段值, 而不是一段 compact 文本。抓取链路(分级反爬、登录态、合规)与 web_fetch 一致。

字段规则示例: {"fields": { "title": {"selector": "h1"}, "price": {"selector": ".price", "regex": "(\d+(?:.\d+)?)", "group": 1}, "main_link": {"selector": "a", "type": "attr", "attr": "href", "abs_url": true}, "link_count": {"selector": "a", "type": "count"}, "tags": {"selector": ".tag", "type": "list", "list_key": "text"}, "items": {"selector": ".item", "type": "list", "fields": { "name": {"selector": ".name"}, "url": {"selector": "a", "type": "attr", "attr": "href", "abs_url": true} }}, "author": {"selector": ".author", "type": "object", "fields": { "name": {"selector": ".name"}, "bio": {"selector": ".bio"} }} }} 字段类型:text(默认,节点归一化文本)/ html / attr(需配 attr)/ count(匹配节点数)/ list(配 fields 返回对象数组,否则按 list_key 取值)/ object(在第一个匹配节点内部继续按 fields 抽取)/ jsonld(不写 selector,用 path 从 JSON-LD 取值,如 {"type": "jsonld", "path": "author.name"})。 后处理:regex + group(正则提取,未命中回落 default)、abs_url(相对链接转绝对)、 strip(去首尾空白,默认 true)、convert(int / float / date,date 输出 ISO 8601)、 required(必填,缺失时整体 ok=false 并返回 missing_required)。

Args: url: 目标网页完整 URL。 schema: 字段抽取规则字典,见上方示例;不传则套用该站 recipe 的 extract 规则。 prompt: 可选的自然语言抽取要求。配置了 LLM(SCRAPE_MCP_LLM_BASE_URL / MODEL)时 交给模型抽取;未配置或调用失败时自动回退确定性 schema,不产生默认外部调用。 字段规则里可用 description 说明字段含义,会一并传给模型。 headers: 自定义请求头;传入时不走共享缓存。 cookies: 自定义 cookie 字典;传入时不走共享缓存。 timeout: 单次请求超时秒数。 deadline_ms: 整次调用的总预算(毫秒),不传表示不限;降级规则同 web_fetch。 wait_for: 仅 L2 生效,等待该 CSS 选择器出现后再取内容;不传则套用站点 recipe。 scroll: 仅 L2 生效,滚动页面触发懒加载;不传则套用站点 recipe。 click: 仅 L2 生效,渲染后点击一次该 CSS 选择器;不传则套用站点 recipe。 bypass_cache: 跳过并绕过响应缓存,强制走网络。 debug: 返回 diagnostics(分层决策与 L2 交互命中情况),不含页面内容。 session: 登录态会话名(默认 default);需与 login 时使用的会话名一致。 proxy: 本次请求使用的代理(如 http://127.0.0.1:7890);不传则按代理池分配, 未配池时沿用全局配置。显式传入代理的请求不读也不写共享缓存。 region: 地区定向(如 us / hk / jp),只在走代理池时生效;没有该地区出口时 直接返回 invalid_argument。会话绑定代理或显式 proxy 优先于它。

NameRequiredDescriptionDefault
urlYes
clickNo
debugNo
proxyNo
promptNo
regionNo
schemaNo
scrollNo
cookiesNo
headersNo
sessionNodefault
timeoutNo
wait_forNo
deadline_msNo
bypass_cacheNo

Output Schema

NameRequiredDescription
resultYes

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations at all, the description carries the full behavioral burden and does so: LLM extraction silently falls back to the deterministic schema and makes no default external call, headers/cookies/explicit proxy bypass the shared cache, region without an exit returns invalid_argument, missing required fields yield ok=false with missing_required, and debug returns diagnostics without page content.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose and the web_fetch contrast are front-loaded, and the long schema-grammar block earns its space because the input schema is an undocumented free-form object. It is still dense and reference-like, with field types, post-processing, and args stacked into one long block rather than separated into scannable sections.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need no explanation, yet the description still calls out the ok=false/missing_required failure shape. For a 15-parameter extraction tool with anti-bot tiers, login state, and LLM fallback, nothing an agent needs to invoke it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% across 15 parameters, and the description compensates for every one of them, including non-obvious ones like session (must match the login session name), proxy (explicit proxy skips cache), deadline_ms (total budget), and bypass_cache. The schema-grammar example with field types (text/html/attr/count/list/object/jsonld) and post-processing (regex+group, abs_url, convert, required) supplies semantics the free-form object schema cannot express.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening sentence gives a specific verb and resource (抓取网页) plus the distinct output contract (按字段 schema 抽取结构化 JSON,字段级数据). It then explicitly contrasts with the sibling web_fetch ("而不是一段 compact 文本"), so an agent can choose between them without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description states the selecting condition ("适合需要'数据而非整页正文'的场景") and names the alternative tool it replaces for full-page text, plus the prompt-vs-schema selection rule with its fallback. It also flags scope limits such as wait_for/scroll/click being L2-only and region only applying through the proxy pool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_fetchA

抓取网页并返回极省 token 的紧凑正文(自动辨别反爬拦截页并如实上报)。

支持 HTML 页面、PDF(需安装 mcp-scrape[pdf];扫描件可开 OCR)、 DOCX/XLSX/PPTX(标准库抽取,无需额外依赖)与纯文本(.txt/.md); 非 HTML 内容直接返回文本,不做 DOM 精简。

Args: url: 目标网页完整 URL,需带 http/https 协议头。 max_tokens: 正文 token 预算上限,超出时保留头尾、省略中段。 link_policy: 链接处理策略。internal=站内链接保留为相对路径、站外降级为纯文本(默认,最省); all=站内外链接全保留;none=全部降级为纯文本。 format: 输出格式。markdown=保留标题/列表/链接标记(默认); text=剥掉 markdown 标记的纯文本;json=按行切分的块数组(content 为字符串列表); chunks=RAG 分块数组(固定字符窗口 + overlap + heading_path,可直接入库)。 chunk_size: format=chunks 时的窗口字符数(默认 1200,范围 1008000)。 chunk_overlap: format=chunks 时相邻块的重叠字符数(默认 200,范围 0chunk_size/2)。 headers: 自定义请求头,如 {"Referer": "...", "User-Agent": "..."};传入时不走共享缓存。 cookies: 自定义 cookie 字典;传入时不走共享缓存。 timeout: 单次请求超时秒数(默认取配置 request_timeout)。 deadline_ms: 整次调用的总预算(毫秒),不传表示不限。预算不够时逐级降级: 压缩单步超时 → 跳过重试(diagnostics 里是 deadline:skip_retry)→ 不升级 L2(deadline:skip_l2)→ 未开始就耗尽时直接返回 timeout。 预算内拿到的正文照常返回,但降级结果不会写入结果缓存。 wait_for: 仅 L2 生效,等待该 CSS 选择器出现后再取内容(适合 SPA);不传则套用站点 recipe。 scroll: 仅 L2 生效,滚动页面触发懒加载/无限滚动;不传则套用站点 recipe。 click: 仅 L2 生效,渲染后点击一次该 CSS 选择器(如"加载更多");不传则套用站点 recipe。 actions: 仅 L2 生效,按顺序执行动作链(click/type/press/select/wait/scroll), 适合"输入关键词 → 点击搜索 → 等待结果"这类交互;单个动作失败只写入 notes, 后续动作继续。传 actions 时会跳过 L1 直接渲染,浏览器不可用时如实提示动作未执行。 bypass_cache: 跳过并绕过响应缓存,强制走网络。 debug: 返回 diagnostics(分层决策、L2 交互命中情况、正文根与候选命中数), 只含计数与决策、不含页面内容;失败时默认也会带精简版决策轨迹。 session: 登录态会话名(默认 default);需与 login 时使用的会话名一致,多身份互不干扰。 proxy: 本次请求使用的代理(如 http://127.0.0.1:7890、socks5://127.0.0.1:1080), L1 与升级后的 L2 都走该出口;不传则按代理池 SCRAPE_MCP_PROXY_POOL 分配, 未配池时沿用 SCRAPE_MCP_PROXY。 显式传入代理的请求不读也不写共享缓存,避免把不同出口的结果混在一起。 region: 地区定向(如 us / hk / jp),只在走代理池时生效:仅从带该地区标签的 出口中选择。池里没有该地区时直接返回 invalid_argument,不会悄悄换出口。 会话已在 SCRAPE_MCP_PROXY_SESSIONS 绑定代理、或显式传了 proxy 时不生效。

NameRequiredDescriptionDefault
urlYes
clickNo
debugNo
proxyNo
formatNomarkdown
regionNo
scrollNo
actionsNo
cookiesNo
headersNo
sessionNodefault
timeoutNo
wait_forNo
chunk_sizeNo
max_tokensNo
deadline_msNo
link_policyNointernal
bypass_cacheNo
chunk_overlapNo

Output Schema

NameRequiredDescription
resultYes

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does so: it discloses caching rules (what bypasses read and write of the shared cache), the deadline_ms degradation ladder with diagnostic tags (deadline:skip_retry, deadline:skip_l2), that degraded results are not cached, that individual action failures only land in notes while later actions continue, and that debug returns counts only, never page content. This is exactly the behavioral context annotations would otherwise have to supply.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose is front-loaded in one sentence and the rest is a clean Args block with one entry per parameter, so length is justified by the 19-parameter surface. Some entries could be trimmed, for example the region entry restates exclusions that are already implied, but there is no filler or marketing language.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 19-parameter fetch tool with an output schema, no annotations, and zero schema coverage, the description covers everything an agent needs: content types, output formats, caching, degradation under budget, session and proxy routing, and failure reporting. Return-value structure is appropriately left to the output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate for all 19 parameters, and it does: every parameter gets meaning, defaults, and where relevant ranges or formats (chunk_size 100~8000, chunk_overlap 0~chunk_size/2, proxy URL examples, header and cookie examples). It also explains non-obvious value semantics such as internal vs all vs none link policies and what each format returns.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening line states a specific verb and resource (fetch a web page and return a token-compact body) and adds a distinguishing behavior: detecting anti-bot block pages and reporting them faithfully. It also enumerates supported content types (HTML, PDF with OCR, DOCX/XLSX/PPTX, plain text). It never names a sibling, so an agent gets no explicit signal about how this differs from web_extract or web_crawl.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Strong parameter-level usage guidance: wait_for/scroll/click/actions are marked 'L2 only', session must match the login session name, region only applies through the proxy pool and returns invalid_argument when unavailable, and explicit proxy or headers/cookies skip the shared cache. What is missing is the routing decision an agent faces most often: when to pick this over web_extract, web_crawl, or web_batch.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_indexA

读取站点的 sitemap,清点页面 URL 并与上次的清单对比。

入口优先取 robots.txt 里的 Sitemap:,没有就回退 /sitemap.xml; sitemap index 会递归展开(最多 20 个文件、2 层),支持 .gz 压缩包。 清单存到 DATA_DIR/index/<host>.json,下次返回新增 / 变更 / 未变 / 移除的计数与 URL。

Args: url: 站点根或任意页面 URL(按 host 找 sitemap);也可直接给 sitemap 地址。 include: 只保留 URL 匹配该正则的条目;不传不过滤。 exclude: 排除 URL 匹配该正则的条目;不传不排除。 limit: 本次最多处理多少条 URL(默认 1000,硬上限 5000)。 check_content: True 时对 lastmod 变过的条目回抓正文并比哈希,内容真没变的不算 changed (单次最多回抓 20 个,避免把站点打穿)。 content_diff: True 时同样回抓正文,但额外与 web_watch 的快照对比并返回 changes(逐 URL 的增删行数与 diff 摘要);快照与 web_watch 共用, 下次单页 web_watch 可直接接续对比。dry_run 时不写快照。 dry_run: True 时只对比、不写入本次清单(便于先预览)。 reset: 丢弃旧清单并重建基线(本次返回 first_seen=true)。 session: 回抓正文时使用的登录态会话名(默认 default)。 deadline_ms: 整次调用的总预算(毫秒),不传表示不限;预算耗尽后不再展开新 sitemap。 proxy: 本次所有请求共用的代理;不传则按代理池分配,未配池时沿用全局配置。

NameRequiredDescriptionDefault
urlYes
limitNo
proxyNo
resetNo
dry_runNo
excludeNo
includeNo
sessionNodefault
deadline_msNo
content_diffNo
check_contentNo

Output Schema

NameRequiredDescription
resultYes

TDQS

A4.3/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does so: sitemap discovery order (robots.txt Sitemap: then /sitemap.xml fallback), recursion caps (20 files, 2 levels), .gz support, the on-disk inventory path, the 20-refetch safeguard against hammering the site, dry_run's no-write semantics, reset discarding the old baseline, deadline exhausting sitemap expansion, and proxy-pool fallback behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

It is front-loaded with a one-line purpose followed by a clean Args block, and each entry carries real information. It is long, but the length is justified by the 11-parameter surface; only minor tightening (e.g. repeating '不传不过滤' style phrasing) would help.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an 11-parameter, stateful, mutating-inventory tool with no annotations, the description covers discovery mechanics, recursion and rate safeguards, persistence location, dry-run/reset state semantics, and cross-tool snapshot sharing. An output schema exists, so the returned counts need not be formally specified, and the description still hints at first_seen/changes shape.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% across 11 parameters, so the description must compensate and it does: every argument is defined with meaning and non-obvious constraints — limit's hard ceiling of 5000, include/exclude as regex over URLs, check_content vs content_diff distinction and their 20-refetch cap, session default, deadline_ms budget semantics, and proxy pool fallback.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The first line states a specific verb+resource: read the site's sitemap, enumerate page URLs, and diff them against the previous inventory. It is clear what the tool does, but it never positions itself against the obvious siblings web_map (site URL mapping) and web_crawl, so the agent must infer the boundary.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied rather than stated: dry_run is described as a preview, reset as rebuilding a baseline, and content_diff is tied to the web_watch snapshot round-trip, which is genuinely useful routing info. However there is no explicit 'use this instead of web_map/web_crawl when…' guidance, and no prerequisites (auth, rate limits) are called out.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_mapA

发现站点 URL:sitemap + 入口页站内链接,可选合并搜索结果。

与 web_index 的区别:web_index 做增量清单对比,web_map 只负责"尽快列出这个站有哪些 可抓的 URL"。无 sitemap 的站点也能从入口页链接得到候选;同一 URL 只出现一次, 并标出来源(sitemap / homepage / search)。

Args: url: 站点根或任意页面 URL;也可直接给 sitemap 地址(此时不再抓入口页链接)。 include: 只保留 URL 匹配该正则的条目;不传不过滤。 exclude: 排除 URL 匹配该正则的条目;不传不排除。 search: 只保留 URL 中匹配该子串的条目(大小写不敏感);不传不过滤。 limit: 最多返回多少条 URL(默认 500,硬上限 5000)。 homepage: True 时抓入口页并提取站内链接(默认 True)。 use_search: True 时额外用 web_search 查 site:<host> 合并结果(默认 False)。 session: 抓入口页时使用的登录态会话名(默认 default)。 deadline_ms: 整次调用的总预算(毫秒),不传表示不限。 proxy: 本次所有请求共用的代理;不传则按代理池分配,未配池时沿用全局配置。

NameRequiredDescriptionDefault
urlYes
limitNo
proxyNo
searchNo
excludeNo
includeNo
sessionNodefault
homepageNo
use_searchNo
deadline_msNo

Output Schema

NameRequiredDescription
resultYes

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does well: it discloses deduplication ('same URL appears only once'), source tagging (sitemap/homepage/search), the default limit of 500 with a hard cap of 5000, proxy-pool allocation behavior, session usage for the entry page, and that use_search triggers an extra web_search call. It stops short of describing failure modes, rate limiting, or required permissions.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose and the sibling distinction are front-loaded before the Args block, and each line earns its place given 10 parameters. The Args list is long but proportionate; only the proxy and deadline descriptions are slightly verbose relative to their importance.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 10-parameter discovery tool, the description plus the existing output schema cover what an agent needs to call it correctly and interpret results. Missing only error/retry behavior and any statement that it performs no mutation, which are minor given the read-only nature of URL discovery.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate, and it does: every one of the 10 parameters gets semantics beyond the schema — include/exclude are regex filters with 'no filter if omitted', search is a case-insensitive substring filter, url may itself be a sitemap address (suppressing entry-page crawling), and defaults/hard caps are spelled out.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource: discover crawlable site URLs via sitemap plus entry-page internal links, optionally merging search results. It explicitly contrasts with the sibling web_index ('web_index does incremental list diff; web_map just lists what URLs this site has'), so an agent can route between the two without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The web_index comparison establishes when to choose this tool, and it covers the fallback case (sites with no sitemap still yield candidates from entry-page links). It does not state explicit when-not conditions or mention other plausible alternatives such as web_crawl or web_search for other purposes, leaving some inference to the agent.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_scheduleA

管理定时监控任务:按间隔执行 web_watch / web_index,变更后推送 webhook。

任务本身不抓取,只记录"盯什么、多久一次、变化后通知谁";实际执行由本机 scrape-mcp schedule daemon 完成(也可用 action=run 手动跑一次)。 首次执行只建立基线,不算变更、不发通知;暂停的任务保留状态但不再被 daemon 执行。 同 kind + url + selector 只有一条任务,重复 create 视为更新间隔与通知地址。

Args: action: list / get / create / pause / resume / delete / run,默认 list。 id: 任务 ID;除 list 与 create 外的动作必填(可从 list 结果拿到)。 kind: watch(对比正文/区块)或 index(对比 sitemap 清单),默认 watch。 url: 目标网页或站点 URL(create 时必填)。 selector: 只看页面内某个区块(CSS 选择器),仅 kind=watch 有效。 interval_s: 执行间隔(秒),默认 3600,允许 60 ~ 604800(一周)。 webhook_url: 检测到变更后 POST 的地址(http/https);不传则只记录状态。 session: 执行时使用的登录态会话名,默认 default。 proxy: 执行时使用的代理;不传则按代理池分配,未配池时沿用全局配置。

NameRequiredDescriptionDefault
idNo
urlNo
kindNowatch
proxyNo
actionNolist
sessionNodefault
selectorNo
interval_sNo
webhook_urlNo

Output Schema

NameRequiredDescription
resultYes

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full behavioral burden and does well: first execution only establishes a baseline (no change, no notification), paused tasks retain state but stop being executed, and changes trigger a webhook POST. It does not say whether delete is permanent or whether action=run blocks, leaving gaps on the more destructive and long-running operations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded prose establishes the concept before the Args block, and the per-parameter lines are dense with no filler. The block is long but justified given nine undocumented parameters; the Args duplication is a necessity, not padding.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a nine-parameter, annotation-free tool with an output schema, the description covers semantics, defaults, constraints, and lifecycle behavior. Remaining gaps are error/permission behavior (e.g., webhook or proxy failures) and whether delete is irreversible.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the Args section is the only documentation — and it covers all nine parameters with defaults, ranges (interval_s 60~604800, default 3600), the valid action list, the watch/index kind values, and conditional rules (id required for everything but list/create; selector only valid for kind=watch). This fully compensates for the empty schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource ("管理定时监控任务") and immediately scopes it: the task records what to watch, how often, and who to notify, dispatching to web_watch / web_index. It also explicitly distinguishes itself from direct-scrape siblings by stating "任务本身不抓取" and naming the daemon that actually executes.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives the default action (list), the manual-execution path (action=run), and the dedupe rule (same kind+url+selector is one task, repeat create = update). What's missing is an explicit statement of when to call web_watch / web_index directly instead of scheduling, which is only implied by the daemon note.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_screenshotA

用真实浏览器渲染页面并截图,保存为 PNG(或 PDF)并返回本地文件路径。

与 web_fetch 共用浏览器池与登录态:需要登录/JS 渲染的页面会先带上已保存的 session,再按需等待某个元素出现后截图,适合"给模型看一眼页面长什么样"。

Args: url: 目标网页完整 URL。 path: 保存路径(可为相对或绝对路径,父目录自动创建);不传则存到 DATA_DIR/screenshots/ 下,文件名按 URL 与参数派生。 full_page: True 时截整页(含滚动区域),默认只截当前视口。 width: 视口宽度(像素);不传时取 device 预设(desktop 1280 / mobile 390)。 height: 视口高度(像素);不传时取 device 预设(desktop 800 / mobile 844)。 wait_ms: 页面加载后额外等待的毫秒数,给动画/懒加载留时间。 selector: 截图前等待该 CSS 选择器出现,命中时按元素包围盒裁剪; 未命中或元素不可见时回退整页并在 notes 中说明。 hide: 截图前隐藏的元素选择器(如固定导航/弹窗);单个失败只写入 notes。 device: 设备预设 desktop / mobile;mobile 使用 390x844 视口、移动 UA 与触屏。 pdf: True 时导出整页 PDF(需 Chromium 内核),此时忽略元素裁剪与 full_page。 actions: 截图前按顺序执行的浏览器动作链(click/type/press/select/wait/scroll), 适合先展开菜单/搜索再截图;单个动作失败只写入 notes,不影响截图。 session: 登录态会话名(默认 default);需与 login 时使用的会话名一致。 deadline_ms: 整次调用的总预算(毫秒),不传表示不限。 proxy: 本次渲染使用的代理;不传则按代理池分配,未配池时沿用全局配置。

NameRequiredDescriptionDefault
pdfNo
urlYes
hideNo
pathNo
proxyNo
widthNo
deviceNo
heightNo
actionsNo
sessionNodefault
wait_msNo
selectorNo
full_pageNo
deadline_msNo

Output Schema

NameRequiredDescription
resultYes

TDQS

A4.5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and delivers: it discloses the return value (local file path), session continuity, per-action and per-selector failure modes that degrade gracefully into `notes`, and the pdf-mode caveat that element cropping and full_page are ignored. Budget and proxy-pool behavior are also spelled out, leaving little an agent would only learn by failing a call.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with a one-sentence purpose plus a short positioning paragraph, then a scannable Args block. It is long, but with 14 parameters almost every line earns its place; only the motivational aside about letting the model see the page is slightly redundant.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return-value detail is partly optional, yet the description still names the returned artifact. For a 14-parameter, multi-modal tool it covers defaults, side effects, and graceful-degradation paths; the chief residual gap is the undocumented schema of individual `actions` entries.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Could not be stronger given constraints: every one of the 14 parameters is explained with defaults and interaction rules (width/height fall back to device presets with the actual 1280x800 / 390x844 values, path defaults to DATA_DIR/screenshots/, selector miss falls back to full page, session must match the login session name). The only weakness is that the `actions` array's item shape is described only by example verbs (click/type/press/select/wait/scroll) rather than a formal structure.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource — render a page in a real browser and screenshot it to PNG/PDF, returning a local file path. It also positions the tool relative to siblings by noting the shared browser pool and login state with web_fetch, so an agent can distinguish it from text-extraction tools without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives a clear use case ("给模型看一眼页面长什么样") and explains that pages requiring login or JS rendering will reuse a saved session, which effectively signals when this tool beats plain fetching. It stops short of explicit exclusions — it never says to use web_fetch/web_extract instead when only text is needed.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

web_watchA

检测网页(或页面内某个区块)自上次调用以来的变化。

每次成功抓取都会保存一份正文快照,下次调用与它对比,返回 changed 与增删行摘要, 适合"每天看一眼这个页面有没有新内容"的场景;无变化时 changed=false 且 diff 为空。 抓取失败(拦截/超时/登录墙)不会覆盖旧快照,等问题解决后仍能对比到真实变化。

Args: url: 目标网页完整 URL。 selector: 只看页面内的某个区块(CSS 选择器),如 "main .article";不传则对比整页正文。 reset: 丢弃该 URL 的旧快照并重建基线(本次返回 first_seen=true)。 bypass_cache: 默认 true,强制拿最新内容;false 时允许命中 CACHE_TTL 内的正文缓存。 session: 登录态会话名(默认 default);需与 login 时使用的会话名一致。 deadline_ms: 整次调用的总预算(毫秒),不传表示不限。 proxy: 本次请求使用的代理;不传则按代理池分配,未配池时沿用全局配置。

NameRequiredDescriptionDefault
urlYes
proxyNo
resetNo
sessionNodefault
selectorNo
deadline_msNo
bypass_cacheNo

Output Schema

NameRequiredDescription
resultYes

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does so well: it discloses that a snapshot is saved per successful fetch, that failures (blocking/timeout/login wall) do not overwrite the old snapshot, that reset rebuilds the baseline and returns first_seen=true, and how caching interacts with bypass_cache. Missing are concurrency/rate-limit expectations and what happens on the very first call for a URL, but the stateful semantics are unusually well surfaced.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose and statefulness are front-loaded, then failure/snapshot behavior, then an Args block that maps one-to-one onto the schema. The Args block is long but each line carries operational meaning; there is no filler or repeated information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 7-parameter stateful tool with no annotations, the definition covers persistence model, failure handling, first-call/baseline behavior, caching, session coupling, and proxy/deadline semantics, and an output schema exists so return shape need not be restated. An agent has enough to call this correctly on the first attempt.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate, and it documents all seven parameters in ways the schema cannot: reset returns first_seen=true, bypass_cache defaults to true and lets CACHE_TTL content be reused, session must match the name used at login, deadline_ms is a total call budget, and proxy falls back to the pool or global config. This adds genuine meaning beyond types and defaults.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: detect changes in a page (or a block within it) since the last call, with snapshot-based diffing. This is clearly a stateful watch tool rather than a plain fetch, so an agent can distinguish it from web_fetch/web_crawl in practice. It never names those alternatives explicitly, which keeps it just short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives a concrete usage scenario ("每天看一眼这个页面有没有新内容") and the conditions that change the behavior for reset and bypass_cache. It also ties session usage to the login tool ("需与 login 时使用的会话名一致"), which is a real routing constraint. No explicit when-not-to-use or alternative-tool guidance is given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 9 tool updatesv0.25.0
    • Changedlogin3 fields changed
      • addedInput schema / properties / cdp_url
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "string"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Cdp Url"
        +}
      • addedInput schema / properties / headless
        Added value: +{
        +  "default": false,
        +  "title": "Headless",
        +  "type": "boolean"
        +}
      • addedInput schema / properties / qr_selector
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "string"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Qr Selector"
        +}
    • Changedweb_batch3 fields changed
      • addedInput schema / properties / chunk_overlap
        Added value: +{
        +  "default": 200,
        +  "title": "Chunk Overlap",
        +  "type": "integer"
        +}
      • addedInput schema / properties / chunk_size
        Added value: +{
        +  "default": 1200,
        +  "title": "Chunk Size",
        +  "type": "integer"
        +}
      • addedInput schema / properties / region
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "string"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Region"
        +}
    • Changedweb_crawl1 field changed
      • addedInput schema / properties / region
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "string"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Region"
        +}
    • Changedweb_extract2 fields changed
      • addedInput schema / properties / prompt
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "string"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Prompt"
        +}
      • addedInput schema / properties / region
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "string"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Region"
        +}
    • Changedweb_fetch3 fields changed
      • addedInput schema / properties / chunk_overlap
        Added value: +{
        +  "default": 200,
        +  "title": "Chunk Overlap",
        +  "type": "integer"
        +}
      • addedInput schema / properties / chunk_size
        Added value: +{
        +  "default": 1200,
        +  "title": "Chunk Size",
        +  "type": "integer"
        +}
      • addedInput schema / properties / region
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "string"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Region"
        +}
    • Changedweb_index1 field changed
      • addedInput schema / properties / content_diff
        Added value: +{
        +  "default": false,
        +  "title": "Content Diff",
        +  "type": "boolean"
        +}
    • Addedweb_schedule
    • Changedweb_screenshot9 fields changed
      • addedInput schema / properties / device
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "string"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Device"
        +}
      • addedInput schema / properties / height / anyOf
        Added value: +[
        +  {
        +    "type": "integer"
        +  },
        +  {
        +    "type": "null"
        +  }
        +]
      • changedInput schema / properties / height / default
        Previous value: -800New value: +null
      • removedInput schema / properties / height / type
        Removed value: -"integer"
      • addedInput schema / properties / hide
        Added value: +{
        +  "anyOf": [
        +    {
        +      "items": {
        +        "type": "string"
        +      },
        +      "type": "array"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Hide"
        +}
      • addedInput schema / properties / pdf
        Added value: +{
        +  "default": false,
        +  "title": "Pdf",
        +  "type": "boolean"
        +}
      • addedInput schema / properties / width / anyOf
        Added value: +[
        +  {
        +    "type": "integer"
        +  },
        +  {
        +    "type": "null"
        +  }
        +]
      • changedInput schema / properties / width / default
        Previous value: -1280New value: +null
      • removedInput schema / properties / width / type
        Removed value: -"integer"
    • Changedweb_search4 fields changed
      • addedInput schema / properties / engines
        Added value: +{
        +  "anyOf": [
        +    {
        +      "items": {
        +        "type": "string"
        +      },
        +      "type": "array"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Engines"
        +}
      • addedInput schema / properties / fallback
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "boolean"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Fallback"
        +}
      • addedInput schema / properties / site
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "string"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Site"
        +}
      • addedInput schema / properties / time_range
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "string"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Time Range"
        +}
  2. 9 tool updatesv0.21.0
    • Addedweb_crawl_cancel
    • Addedweb_crawl_list
    • Addedweb_crawl_start
    • Addedweb_crawl_status
    • Changedweb_fetch1 field changed
      • addedInput schema / properties / actions
        Added value: +{
        +  "anyOf": [
        +    {
        +      "items": {
        +        "additionalProperties": true,
        +        "type": "object"
        +      },
        +      "type": "array"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Actions"
        +}
    • Addedweb_index
    • Addedweb_map
    • Addedweb_screenshot
    • Addedweb_search
  3. 1 tool updatev0.19.0
    • Addedweb_watch
  4. 8 tool updatesv0.10.1
    • Changedlogin1 field changed
      • addedInput schema / properties / session
        Added value: +{
        +  "default": "default",
        +  "title": "Session",
        +  "type": "string"
        +}
    • Addedsession_export
    • Addedsession_import
    • Addedsession_list
    • Changedweb_batch9 fields changed
      • addedInput schema / properties / bypass_cache
        Added value: +{
        +  "default": false,
        +  "title": "Bypass Cache",
        +  "type": "boolean"
        +}
      • addedInput schema / properties / cookies
        Added value: +{
        +  "anyOf": [
        +    {
        +      "additionalProperties": {
        +        "type": "string"
        +      },
        +      "type": "object"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Cookies"
        +}
      • addedInput schema / properties / deadline_ms
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "integer"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Deadline Ms"
        +}
      • addedInput schema / properties / debug
        Added value: +{
        +  "default": false,
        +  "title": "Debug",
        +  "type": "boolean"
        +}
      • addedInput schema / properties / format
        Added value: +{
        +  "default": "markdown",
        +  "title": "Format",
        +  "type": "string"
        +}
      • addedInput schema / properties / headers
        Added value: +{
        +  "anyOf": [
        +    {
        +      "additionalProperties": {
        +        "type": "string"
        +      },
        +      "type": "object"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Headers"
        +}
      • addedInput schema / properties / proxy
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "string"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Proxy"
        +}
      • addedInput schema / properties / session
        Added value: +{
        +  "default": "default",
        +  "title": "Session",
        +  "type": "string"
        +}
      • addedInput schema / properties / timeout
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "number"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Timeout"
        +}
    • Addedweb_crawl
    • Changedweb_extract16 fields changed
      • addedInput schema / properties / bypass_cache
        Added value: +{
        +  "default": false,
        +  "title": "Bypass Cache",
        +  "type": "boolean"
        +}
      • addedInput schema / properties / click
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "string"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Click"
        +}
      • addedInput schema / properties / cookies
        Added value: +{
        +  "anyOf": [
        +    {
        +      "additionalProperties": {
        +        "type": "string"
        +      },
        +      "type": "object"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Cookies"
        +}
      • addedInput schema / properties / deadline_ms
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "integer"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Deadline Ms"
        +}
      • addedInput schema / properties / debug
        Added value: +{
        +  "default": false,
        +  "title": "Debug",
        +  "type": "boolean"
        +}
      • addedInput schema / properties / headers
        Added value: +{
        +  "anyOf": [
        +    {
        +      "additionalProperties": {
        +        "type": "string"
        +      },
        +      "type": "object"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Headers"
        +}
      • addedInput schema / properties / proxy
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "string"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Proxy"
        +}
      • removedInput schema / properties / schema / additionalProperties
        Removed value: -true
      • addedInput schema / properties / schema / anyOf
        Added value: +[
        +  {
        +    "additionalProperties": true,
        +    "type": "object"
        +  },
        +  {
        +    "type": "null"
        +  }
        +]
      • addedInput schema / properties / schema / default
        Added value: +null
      • removedInput schema / properties / schema / type
        Removed value: -"object"
      • addedInput schema / properties / scroll
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "boolean"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Scroll"
        +}
      • addedInput schema / properties / session
        Added value: +{
        +  "default": "default",
        +  "title": "Session",
        +  "type": "string"
        +}
      • addedInput schema / properties / timeout
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "number"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Timeout"
        +}
      • addedInput schema / properties / wait_for
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "string"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Wait For"
        +}
      • changedInput schema / required
        Previous value: -[
        -  "url",
        -  "schema"
        -]New value: +[
        +  "url"
        +]
    • Changedweb_fetch12 fields changed
      • addedInput schema / properties / bypass_cache
        Added value: +{
        +  "default": false,
        +  "title": "Bypass Cache",
        +  "type": "boolean"
        +}
      • addedInput schema / properties / click
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "string"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Click"
        +}
      • addedInput schema / properties / cookies
        Added value: +{
        +  "anyOf": [
        +    {
        +      "additionalProperties": {
        +        "type": "string"
        +      },
        +      "type": "object"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Cookies"
        +}
      • addedInput schema / properties / deadline_ms
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "integer"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Deadline Ms"
        +}
      • addedInput schema / properties / debug
        Added value: +{
        +  "default": false,
        +  "title": "Debug",
        +  "type": "boolean"
        +}
      • addedInput schema / properties / format
        Added value: +{
        +  "default": "markdown",
        +  "title": "Format",
        +  "type": "string"
        +}
      • addedInput schema / properties / headers
        Added value: +{
        +  "anyOf": [
        +    {
        +      "additionalProperties": {
        +        "type": "string"
        +      },
        +      "type": "object"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Headers"
        +}
      • addedInput schema / properties / proxy
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "string"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Proxy"
        +}
      • addedInput schema / properties / scroll
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "boolean"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Scroll"
        +}
      • addedInput schema / properties / session
        Added value: +{
        +  "default": "default",
        +  "title": "Session",
        +  "type": "string"
        +}
      • addedInput schema / properties / timeout
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "number"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Timeout"
        +}
      • addedInput schema / properties / wait_for
        Added value: +{
        +  "anyOf": [
        +    {
        +      "type": "string"
        +    },
        +    {
        +      "type": "null"
        +    }
        +  ],
        +  "default": null,
        +  "title": "Wait For"
        +}
  5. 1 tool updatev0.2.0
    • Addedweb_extract
  6. 3 tool updatesv0.1.0
    • First observedlogin
    • First observedweb_batch
    • First observedweb_fetch

TDQS

A4.1/5.0

Scored across 18 tools

Disambiguation4/5

Most tools target clearly distinct operations (fetch vs extract vs screenshot vs search vs sessions), and descriptions explicitly contrast similar tools like web_index and web_map. However, web_crawl and web_crawl_start overlap in purpose (sync vs async), and login vs session_import both establish login state, creating some selection ambiguity.

Naming Consistency4/5

The set largely follows a web_<operation> convention with a consistent session_* family for session management, making names predictable. The main deviations are login (not session_-prefixed) and a mix of noun-style names like web_batch alongside verb-style names like web_fetch.

Tool Count4/5

18 tools for a full-featured scraping server is slightly heavy but broadly justified by the breadth of capabilities: fetch, extract, crawl, async jobs, search, screenshots, sitemap indexing, monitoring, and sessions. A few tools, especially the synchronous and asynchronous crawl variants, could be consolidated.

Completeness5/5

The surface covers the scraping lifecycle comprehensively: single and batch fetching, structured extraction, crawling (sync and async), URL discovery via sitemap and links, search, screenshots, change monitoring, scheduling, and session/cookie management. No obvious functional gaps remain for the stated domain.

Maintenance

ActivityMaintained
ResponsivenessNo recent community issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    C
    maintenance
    Enables AI agents to read web pages reliably, returning clean markdown content, hyperlinks, and metadata without navigation or ad noise.
    3
    10 npm
    MIT
  • A
    license
    A
    quality
    Not graded
    maintenance
    Enables AI agents to extract clean, structured web content (articles, tables, links, visual layouts) optimized for LLM token efficiency, with fast response times and optional JavaScript support.
    5
    56 npm
    MIT