universal-scraper
Enables scraping Taobao search and listing data through a logged-in browser session, with login-state persistence and anti-bot handling.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@universal-scraperscrape the product listings from example.com"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
universal-scraper
把“从这个网站拿数据”变成一句人话。
配置驱动 · 四级反爬降级 · 数据校验 · 86 个 Python 模块 · 21 个站点配方 · 738 项测试 · MIT

30 秒验证它能跑
git clone https://github.com/qqkkrr/universal-scraper.git
cd universal-scraper
pip install -r requirements.txt
bash scripts/setup.sh # 装依赖 + 环境体检
python -m universal_scraper.cli doctor
python -m universal_scraper.cli fetch "https://example.com"看到 doctor 的“22/23 项通过”和 fetch 的 200 响应,就说明引擎跑通了。

中文文档:安装与使用说明 · 使用说明书 · 使用者解决方案手册
Related MCP server: pulldown
它解决什么问题
需要的数据散在不同网站上:有的页面直接给,有的藏在接口里,有的要登录才看得到。人工复制粘贴慢,而且同一站点换一天再采,字段和口径可能就变了,结果不可复现。
现成的采集工具大多假设站点结构稳定、没有防护。真实情况恰好相反:同一个任务里,A 站一次就能拿到,B 站要过三道关。所以真正难的不是“发请求”,而是判断这一站属于哪一类,再选一条成本最低、又不越界的路线。
怎么工作
把“抓数据”拆成五步,每步有明确的通过条件,前一步不过就不进下一步。
步骤 | 做什么 | 通过条件 |
01 判型 | 轻量请求一次目标页,判断是静态页、JS 壳,还是数据在接口里 | 能说出这站属于哪一类 |
02 选路线 | 按反爬强度选层级,能走低层不走高层 | 取数成本最低 |
03 抓小样 | 先抓 5–10 条,字段摆出来给人看 | 字段对得上再全量 |
04 全量 | 限速、断点续跑、异常隔离 | 不中断、不重复 |
05 复核 | 字段完整率、唯一键、同页双抓比对 | 完整率 ≥ 90% |
反爬的四级降级链
不预设“这站一定很强”,逐级试探,遇到拦截才升级。多数站点停在 L1,真正需要打开浏览器的不到两成。
层级 | 手段 | 适用 |
L1 | HTTP 直连,带浏览器指纹与必要请求头 | 静态页、公开接口 |
L2 | 浏览器指纹模拟,绕开简单环境检测 | 轻中度防护 |
L3 | 无头浏览器渲染,等数据加载后取 DOM 或接口响应 | JS 渲染的壳页 |
L4 | 复用使用者本人已登录的浏览器会话 | 登录后才可见的内容 |
L4 只做“复用登录态”:由使用者本人完成登录,工具不代填密码、不代过验证码。
验证到什么程度
判断标准只有一条:换一个站点,这套流程还成立吗。
类型 | 代表 | 难点 |
司法公开 | 中国裁判文书网 | 登录墙 + 强风控 |
政务服务 | 12306 | 会话校验 |
统计口径 | 国家统计局 | 多维查询、翻页结构 |
地方公开 | 各级政府门户 | 页面陈旧、编码混乱 |
仓库内置 21 个站点配方,覆盖公开 API、SSR、JS 壳和需登录调优四类:

python -m universal_scraper.cli sites引擎本体经 10 轮审计,当前 pytest 收集 738 项测试;77 个测试文件,另有独立 CI 脚本。
一个真实例子
采 A 股「算力 / 智算中心 / 东数西算 / 数据中心」相关公告时,标题里带 HTML 高亮标签,直接统计会把「计算机能力」「电力」误判成「算力」。按连续词过滤后剔除 402 条,671 条收敛为 269 条——如果不做这一步,结论会虚高 150%。
边界
只采公开数据。不绕过登录墙获取非公开内容,不逆向签名算法,不破解付费/版权墙。
默认单站每秒不超过 1 次请求;大任务开工前先估算总请求数。
站点改版会让配置失效。工具能降低维护成本,但消不掉它——所以判型表和校验流程是核心资产,不是附件。
数据是否可信、口径是否一致,仍要人来判断。工具负责把可疑处标出来,不替人下结论。
目录
路径 | 内容 |
SKILL.md | AI 读取的工作流程(五幕引导 + 铁律) |
universal_scraper/ | 采集引擎本体(86 个 Python 模块) |
references/ | 反爬升级手册 / 配置字段 / 站点配方 |
scripts/ | 环境安装、体检、调试 Chrome 启动器、浏览器桥 |
challenges/ | 挑战用例 |
webui/ | 本地可视化界面(cli webui) |
docs/zh/ | 中文使用手册、安装说明、审查记录 |
docs/assets/ | README 图片与 GIF |
CLI 速览
fetch(侦察)· run(按配置/任务采集,支持断点续跑)· verify(结果复核)· snapshot(页面快照离线解析)· batch(批量队列)· pdf(PDF 下载与表格)· compliance(开工前合规预检)· budget(限速与封锁台账)· research(科研批量采集)
共 49 个 CLI 子命令,查看完整列表:
python -m universal_scraper.cli --help作为 AI 技能安装
把仓库放进技能目录:
git clone https://github.com/qqkkrr/universal-scraper.git \
~/.agents/skills/universal-scraper
bash ~/.agents/skills/universal-scraper/scripts/setup.sh之后直接用日常语言让 AI 干活,例如:
“把这个列表页的商品都抓下来,要名称、价格、链接,存桌面”
“帮我下载这 10 篇论文的 PDF”
“这个网站要登录,我扫码登录一次之后你接着抓”
“盯着这个页面,有更新就告诉我”
License
MIT
Available Tools
8 toolsautoA
用一句话描述爬虫任务,AI 自动生成配置、自动运行、失败自修复,返回结构化结果。适合:不会写选择器/配置的非技术用户。
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | 最多抓取条数(可选) | |
| rounds | No | 自动重试轮数(默认 2,失败会自修复配置) | |
| description | Yes | 中文/英文任务描述,如:抓取 https://quotes.toscrape.com/ 的名言、作者和标签,翻 2 页 |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden. It usefully discloses the multi-stage pipeline (config generation, execution, failure self-repair) and that output is structured, but says nothing about latency, cost, terminal-failure behavior when self-repair is exhausted, or whether any state is persisted.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two compact sentences: the capability pipeline is front-loaded, then the audience qualifier. Nothing is redundant and no sentence restates the schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With three simple parameters and full schema coverage, the input side is well covered. However, there is no output schema, so the vague claim "返回结构化结果" leaves the shape of the returned data undefined for an agent that must decide whether the result is usable — a gap that should have been spelled out.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all three parameters (description, limit, rounds) are already documented in the schema, including the retry semantics of rounds and an example task string. The description adds no format or syntax guidance beyond what the schema already supplies, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb chain (auto-generate config → auto-run → self-repair) and resource (crawler task) plus a structured return, which is far more specific than the ambiguous name 'auto'. It implicitly distinguishes itself from config-driven siblings like scrape/crawl by targeting users who "不会写选择器/配置", but it never names those siblings explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives an explicit target-user condition — "适合:不会写选择器/配置的非技术用户" — which tells the agent this is the zero-config entry point. It does not name alternatives (scrape, crawl, extract) or state when NOT to use this tool, so routing still requires the agent to infer from sibling names.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
booksA
图书目录采集:按 ISBN 清单抓取豆瓣评分/作者/出版社、京东+当当聚合比价,导出 booklist.csv / booklist.md / books.json / crawl_log.md。只采集书目数据与公开封面,不下载电子书/PDF,不绕过登录或付费墙;0 条/部分失败会返回逐项诊断与行动方案(status=NO_DATA/PARTIAL),不会伪装成功。
| Name | Required | Description | Default |
|---|---|---|---|
| out | No | 输出目录(默认 outputs/book_catalog;相对路径基于服务启动目录) | |
| spec | No | 内联 spec:{"name":"我的书单","books":[{"title":"书名","isbn":"9787563394180"}]} | |
| interval | No | 每个 HTTP 请求间隔秒数(默认 1.0,防限流) | |
| spec_path | No | spec JSON 文件路径(与 spec 二选一;相对路径按本仓库根解析,如 examples/books.spec.json) | |
| download_covers | No | 是否下载公开封面(默认 true) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden, and it does well: it discloses the failure contract (status=NO_DATA/PARTIAL with per-item diagnostics and an action plan) and explicitly says it will not fake success. It also constrains scope and repeats that login/paywalls are not bypassed. It stops short of stating auth requirements or the mechanism behind the interval-based throttling.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
It leads with the core action and then lists the outputs, and every clause in the one-block description earns its place: scope, exclusion, exports, and failure semantics. It is dense but not padded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
There is no output schema, yet the description names the exact returned files and the NO_DATA/PARTIAL status values, so an agent can anticipate the return shape. Combined with schema-covered parameters, an agent has nearly everything needed to call it, though it does not describe the spec JSON contract or how covers are stored.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all five parameters, including the nested spec object, are already documented in the schema. The description adds domain color (comparing JD and Dangdang, exporting booklist.csv/md/json) but no syntax or format guidance beyond what the schema provides, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description names a specific verb and resource: it collects book-catalog data from an ISBN list (Douban ratings, author, publisher, JD+Dangdang price comparison) and exports four named file types. That detail differentiates it sharply from generic siblings such as scrape, crawl, and bulk_scrape, which do not mention books or export formats.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It states the input shape context (an ISBN list) and the output artifacts, plus clear scope exclusions: no e-book/PDF downloading and no login or paywall bypass. It does not explicitly say when to prefer this over scrape or bulk_scrape, so the sibling routing is implied rather than stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
bulk_scrapeA
批量抓取多个 URL(一次调用替代 N 次往返;逐 URL 隔离失败——单个 URL 出错不影响其余)。默认最多 10 个、上限 30(防爆)。每个 URL 有间隔(默认 1s)保持礼貌;返回每条的 url/status/摘要。
| Name | Required | Description | Default |
|---|---|---|---|
| max | No | 最多处理几个(默认 10,上限 30) | |
| urls | Yes | URL 列表(最多 30 个) | |
| article | No | 正文抽取(默认 false) | |
| interval | No | 每个 URL 间隔秒数(默认 1.0) | |
| selector | No | 只取匹配元素的文本(可选) | |
| max_chars | No | 每条摘要上限字符(默认 8000) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations the description carries the full burden, and it does substantial work: per-URL failure isolation ('one URL error doesn't affect the rest'), a default of 10 and hard cap of 30, and a politeness interval. It stops short of explaining what happens past the cap (truncation vs. error), per-request timeouts, or total runtime for a 30-URL batch.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Four tightly packed clauses, front-loaded with the core value proposition and then the safety limits and return shape. Each clause carries distinct information (batch economics, failure isolation, caps, politeness, output) with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 6-parameter batch tool with no output schema and no annotations, the description covers the essential behaviors: batching, isolation, caps, throttle, and the per-item return fields (url/status/summary). It leaves the truncation behavior at the 30-cap and timeout semantics unspecified, which is a modest but real gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so every parameter already carries its own doc text and the baseline is 3. The description only repeats the max/interval defaults and adds the return-field shape; it adds no syntax or edge-case meaning (e.g. selector precedence vs article extraction) beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description states a specific verb and resource (搬运抓取多个 URL) and immediately differentiates itself from the single-URL sibling 'scrape' by framing the value as 'one call replaces N round trips'. An agent can tell this apart from scrape/crawl without opening either schema.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives clear context for when to use it (many URLs, replacing N round trips) and implies the alternative to looping 'scrape'. However it does not explicitly name 'scrape' or 'crawl' or state when NOT to use bulk mode (e.g. for a single URL or full-site traversal), so the routing guidance stays implicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
checkB
引擎体检:返回版本、已通过测试(100+100+27)、能力矩阵、LLM 可用性、目录结构。
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries the full behavioral burden, and it leaves real questions open: whether calling it merely reads state or actively re-runs the test suites (potentially slow or costly), whether it is side-effect free, and whether any configuration is required. It does enumerate the return payload, which is its one behavioral contribution, but that is thin for an unannotated tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
A single compact sentence that front-loads the core concept (engine health check) and then lists the payload items. No filler, no repetition of the tool name, and every clause carries information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description must explain the return values, and it does so by naming all five result categories, which is the critical piece for a zero-parameter diagnostic. What is missing is the operational framing — whether this is a cheap read or a heavier self-test — so it falls just short of fully complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool takes zero parameters, so there is nothing for the description to disambiguate beyond what the empty schema already states. Baseline 4 applies; the description correctly avoids inventing parameter discussion.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description frames the tool as an 'engine health check' (引擎体检) and enumerates exactly what comes back: version, passed tests, capability matrix, LLM availability, directory structure. That is a concrete, recognizable diagnostic purpose, and it clearly differs from the scraping-oriented siblings (scrape, crawl, screenshot). It stops short of explicitly naming when to prefer it over those siblings, but the intent is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
There is no guidance on when to call this tool: no mention of using it as a preflight/health-check step, no exclusions, no reference to alternatives among the sibling tools. An agent knows what it returns but not under what circumstances invoking it is appropriate or wasteful.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
crawlA
从入口 URL 递归爬取整站/子目录,返回每页标题+正文+链接。适合:整站抓取、文档站镜像。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | 入口网址 | |
| deny | No | 跳过匹配该正则的链接 | |
| allow | No | 只爬匹配该正则的链接(如 ^/docs/) | |
| depth | No | 递归深度(默认 2) | |
| browser | No | 是否浏览器渲染(默认 false) | |
| max_pages | No | 最多抓取页数(默认 100) | |
| same_domain | No | 只爬同域名(默认 false) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden: it does disclose the return shape and the recursive scope, which is useful. However, for a potentially heavy crawl it says nothing about rate limiting, timeouts, robots/politeness, or what happens when a page fails, and it never flags that same_domain defaults to false so the crawl may leave the domain.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two sentences, zero waste, with the core action and scope front-loaded before the use-case hint. Every clause earns its place.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no output schema, the description usefully states the per-page return fields, and all seven parameters are covered by the schema. The remaining gap is behavioral: cost/duration and rate-limit expectations for a recursive crawler are unstated.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100% and the parameter descriptions are good (regex example '^/docs/', defaults for depth/browser/max_pages/same_domain), so the schema does the heavy lifting. The description adds nothing parameter-specific beyond the general notion of recursion, so the baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a specific verb and resource: recursively crawl a whole site/subdirectory starting from an entry URL, and it also names the return shape (title + body + links per page). The word '递归' plus '整站/子目录' implicitly separates it from the single-page scrape sibling, but no sibling is named explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
'适合:整站抓取、文档站镜像' gives positive use cases, which is more than nothing. But it never says when NOT to use it, nor points to alternatives such as scrape for a single page or bulk_scrape for a known URL list, so the agent must infer the boundary from sibling names.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
extractA
把 HTML 源码转成干净 Markdown/正文/表格,用于省 token 地阅读网页内容。默认先净化(剥离隐藏元素/注释/零宽字符等提示注入载体)再转换;main_only=true 时额外收窄到主内容容器(更适合阅读文章页)。
| Name | Required | Description | Default |
|---|---|---|---|
| raw | No | 跳过净化(默认 false;仅在确需原始 HTML 时用) | |
| html | Yes | HTML 源码 | |
| mode | No | markdown(默认)/ article / table | |
| base_url | No | 用于把相对链接转绝对(可选) | |
| main_only | No | 只取主内容容器(默认 false;正文页建议 true) | |
| max_chars | No | 输出上限字符(默认 50000,防爆 token) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the burden and does well: it discloses that sanitization runs by default stripping hidden elements, comments, and zero-width characters (prompt-injection vectors), and that main_only narrows to the main content container. It does not cover behavior on malformed HTML or that the operation is a side-effect-free pure transform.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two dense sentences with the transformation stated first and defaults/options after; minimal waste. Slightly clause-heavy but every clause carries information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a one-required-parameter transform with no output schema and no annotations, it covers purpose, defaults, sanitization behavior, and output kinds. Missing only return-shape specifics and failure behavior, which are minor here.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 100%, so all six parameters are already documented. The description mainly reinforces the sanitization/raw interaction and main_only intent that the schema already states, adding only marginal nuance beyond the structured fields.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States a concrete transformation: converting HTML source into clean Markdown/body text/tables, with the goal of token-efficient reading. The input being raw HTML rather than a URL implicitly distinguishes it from scrape/crawl siblings, though no sibling is named explicitly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Gives conditional guidance (main_only=true is better for article pages; raw only when truly needed) but never states when to pick this tool over scrape, crawl, or auto. Usage is implied by the input type rather than spelled out.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
scrapeA
一键抓取一个 URL,返回干净文本/Markdown/正文/表格/链接。适合:读网页内容、取正文、找表格、收集链接。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | 目标网址(http/https) | |
| links | No | 同时返回页面里的链接(默认 false) | |
| proxy | No | 代理地址(可选) | |
| table | No | 提取页面里的表格为 JSON(默认 false) | |
| article | No | 只提取正文(默认 false) | |
| browser | No | 是否用浏览器渲染 JS 页面(默认 false) | |
| selector | No | CSS 选择器,只返回匹配内容(如 h1、.content) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description must carry the full behavioral burden. It describes output formats and use cases, but omits important scraping behavior such as rate limits, authentication needs, anti-bot handling, failure modes, and when JS rendering is required.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded with the core action and output formats, followed by concrete use cases. Every phrase contributes to helping the agent understand the tool without unnecessary length.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given seven parameters, no annotations, and no output schema, the description is adequate but thin. It lists possible return content types, but does not explain the response structure, errors, or when specific options like selector, proxy, or browser should be used.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already documents all seven parameters. The description adds only a high-level output summary and does not provide additional syntax or semantics beyond the schema baseline.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: scrape a single URL, returning clean text/Markdown/article/tables/links. This is clear, but it does not explicitly distinguish the tool from siblings such as crawl, bulk_scrape, or extract.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It lists clear usage contexts: reading webpage content, extracting article text, finding tables, and collecting links. However, it gives no exclusions or explicit guidance on when to prefer crawl, bulk_scrape, screenshot, or extract instead.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
screenshotA
对页面截图(浏览器渲染)并保存为图片文件,返回文件路径。用于人工核对渲染结果/排障;图片本身不内联返回。
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | 目标网址(http/https) | |
| path | No | 保存路径(可选;默认 outputs/mcp_shots/<时间戳>.png) | |
| full_page | No | 整页截图(默认 false=视口) | |
| wait_selector | No | 等该元素出现后再截图(可选) |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full behavioral burden. It helpfully discloses that the page is browser-rendered, the image is saved to a file, the file path is returned, and the image is not returned inline, but it omits operational details such as overwrite behavior, permissions, timeouts, or other side effects of writing a file.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded, first stating the action and output, then the intended use. Every clause adds useful information with no redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given a fully described four-parameter schema and no output schema, the description covers the essential return behavior by stating that the file path is returned and the image is not inline. It is nearly complete for this tool, though it could say more about file handling or constraints.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so the schema already explains url, path, full_page, and wait_selector in detail. The description adds no parameter-specific meaning beyond what the schema provides, so the baseline of 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb and resource: screenshot a page via browser rendering, save it as an image file, and return the file path. This is clear enough to distinguish it from scraping or crawling siblings by its image-file output, although it does not explicitly name or contrast those siblings.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives a clear usage context: manual verification of rendering results and troubleshooting. It does not specify when not to use it or explicitly compare it with alternatives such as scrape or crawl, so it stops short of full routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
8 tool updates
v2.2.0- First observed
auto - First observed
books - First observed
bulk_scrape - First observed
check - First observed
crawl - First observed
extract - First observed
scrape - First observed
screenshot
TDQS
Scored across 8 tools
Most tools have distinct roles (crawl, bulk_scrape, screenshot, check, books), but scrape and extract overlap heavily because both turn web content into clean Markdown/text/tables, differing mainly by URL vs HTML input. The auto tool also overlaps conceptually with scrape/crawl by generating and running scraping tasks, so agents may hesitate between auto, scrape, and extract.
Names mix single verbs (scrape, crawl, check, extract), a noun-like tool (books), a vague automation label (auto), and one snake_case compound (bulk_scrape). The set is readable but has no predictable verb_noun or resource_action convention.
Eight tools is well-scoped for a universal scraper, covering single-page scraping, batch scraping, recursive crawling, extraction, screenshots, diagnostics, AI-assisted automation, and a book-data specialization. Each tool has a plausible place in the workflow.
The surface covers the core scraping lifecycle: fetch one page, fetch many, crawl a site, extract from HTML, screenshot, diagnose, and automate configuration. Minor gaps exist around explicit session/cookie/proxy management or structured job status, but the main workflows are covered.
Maintenance
Related MCP Connectors
Web scraping MCP server: any URL to Markdown or JSON, with JavaScript rendering and batch jobs.
Web scraping MCP server — scrape, extract structured data, screenshot any site with anti-bot bypass.
One MCP server for all web data. Scrape, search, crawl, and AI research in one install.
One MCP server for 180+ live web-data APIs returning clean JSON from sites that block scrapers.
Related MCP Servers
- AlicenseAqualityBmaintenanceMCP server for web scraping — extract clean markdown, links, and metadata from any URL. Free Firecrawl alternative.562 npm6MIT
- AlicenseNot gradedqualityCmaintenanceMCP server for retrieving web pages as clean Markdown, with configurable detail levels and optional Chromium rendering for JavaScript-heavy pages.1MIT
- AlicenseNot gradedqualityDmaintenanceMCP server to scrape web pages to clean Markdown via headless Chromium, with support for single or batch URLs.MIT
- MIT