Skip to main content
Glama
guanyuyan
by guanyuyan

web_crawl

Discover and batch-crawl same-site links from a start page, or follow "Next" pagination, to collect the first N articles with per-page failure isolation and compliance checks.

Instructions

从起始页发现站内链接并批量抓取,或按"下一页"顺序翻页,适合把一个栏目的前 N 篇读进来。

每页都走完整的分级抓取与合规链路(robots/黑白名单/限流/同域并发), 单页失败只影响该页,不中断整次爬取;静态资源链接(图片/压缩包等)自动跳过。

两种模式:

  • 链接模式(默认):按 depth 分层发现站内链接,limit 控制总页数。

  • 分页模式:传 next_selector(或站点 recipe 配置),顺序跟进"下一页",max_pages 控制页数。

Args: url: 起始页 URL。 depth: 链接模式的跳转层数。0=只抓起始页;1=再抓起始页上的链接(默认);最大 3。 limit: 链接模式最多抓取的页面总数(含起始页),受配置 crawl_max_pages 上限约束。 same_host: 只跟进同主机链接(默认 True)。 max_tokens: 每页正文 token 预算上限。 link_policy: 同 web_fetch。 next_selector: 分页模式:指向"下一页"的 CSS 选择器;有值时忽略 depth。 max_pages: 分页模式最多翻多少页(默认 5),受 crawl_max_pages 上限约束。 headers: 所有页面共用的自定义请求头;传入时不走共享缓存。 cookies: 所有页面共用的自定义 cookie;传入时不走共享缓存。 timeout: 单次请求超时秒数。 deadline_ms: 整次爬取的总预算(毫秒),不传表示不限。预算耗尽后不再抓新页, 已抓到的页面照常返回;排队中的子任务直接返回 timeout。 bypass_cache: 跳过并绕过响应缓存。 session: 登录态会话名(默认 default);对本次所有页面生效。 proxy: 本次爬取所有页面共用的代理;不传则按代理池分配,未配池时沿用全局配置。 显式传入代理时不走共享缓存。 ctx: MCP 上下文,由框架注入,用于逐页上报进度(客户端请求进度时才发通知)。

Input Schema

NameRequiredDescriptionDefault
urlYes
depthNo
limitNo
proxyNo
cookiesNo
headersNo
sessionNodefault
timeoutNo
max_pagesNo
same_hostNo
max_tokensNo
deadline_msNo
link_policyNointernal
bypass_cacheNo
next_selectorNo

Output Schema

NameRequiredDescriptionDefault
resultYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. Addedv0.10.1

TDQS

A4.2/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does so well: every page goes through the full compliance chain (robots/allow-block list/rate limiting/same-host concurrency), a single page failure does not abort the crawl, static asset links are skipped automatically. It also discloses non-obvious side effects: explicit headers/cookies/proxy bypass the shared cache, and when deadline_ms is exhausted already-fetched pages are still returned while queued subtasks return timeout.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose and mode split are front-loaded, then a structured Args list follows — the bulk is necessary given zero schema descriptions. The trailing ctx entry is framework plumbing that adds little for an agent, a small amount of dead weight.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 15-parameter crawler with no annotations, the description covers mode selection, failure isolation, caching, deadlines, and compliance — enough to call it correctly. Return values are covered by the existing output schema, but auth/session prerequisites and concurrency limits are only gestured at.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the Args block is doing all the work — and it documents 14 of 15 parameters with real semantics: depth range 0–3, limit bounded by crawl_max_pages, next_selector suppressing depth, max_pages default 5, session applying to all pages. It loses a point because a few details (e.g. timeout/proxy fallback nuances vs. global config) remain terse, but the compensation for a 15-param, 0%-covered schema is substantial.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening states a specific verb and resource: discover in-site links from a start page and batch crawl them, or page through via "next page". It further splits the tool into two named modes (link mode / pagination mode) so the agent understands the tool's two distinct behaviors. It only references a sibling indirectly ("link_policy: 同 web_fetch") rather than explicitly contrasting scope with web_fetch/web_batch, so it stops short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

"适合把一个栏目的前 N 篇读进来" gives a concrete selection scenario, and the two-mode block states which parameter selects which mode (next_selector overriding depth). There is no explicit "when not to use this / use web_fetch for a single page instead" exclusion, which keeps it at 4 rather than 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.