Skip to main content
Glama
scrape-mcp
by guanyuyan

web_crawl

Crawl a start page to discover and batch-fetch internal links or follow next-page selectors, enabling ordered collection of the first N articles from a section with per-page compliance and failure isolation.

Instructions

从起始页发现站内链接并批量抓取,或按"下一页"顺序翻页,适合把一个栏目的前 N 篇读进来。

每页都走完整的分级抓取与合规链路(robots/黑白名单/限流/同域并发), 单页失败只影响该页,不中断整次爬取;静态资源链接(图片/压缩包等)自动跳过。

两种模式:

  • 链接模式(默认):按 depth 分层发现站内链接,limit 控制总页数。

  • 分页模式:传 next_selector(或站点 recipe 配置),顺序跟进"下一页",max_pages 控制页数。

Args: url: 起始页 URL。 depth: 链接模式的跳转层数。0=只抓起始页;1=再抓起始页上的链接(默认);最大 3。 limit: 链接模式最多抓取的页面总数(含起始页),受配置 crawl_max_pages 上限约束。 same_host: 只跟进同主机链接(默认 True)。 max_tokens: 每页正文 token 预算上限。 link_policy: 同 web_fetch。 next_selector: 分页模式:指向"下一页"的 CSS 选择器;有值时忽略 depth。 max_pages: 分页模式最多翻多少页(默认 5),受 crawl_max_pages 上限约束。 headers: 所有页面共用的自定义请求头;传入时不走共享缓存。 cookies: 所有页面共用的自定义 cookie;传入时不走共享缓存。 timeout: 单次请求超时秒数。 deadline_ms: 整次爬取的总预算(毫秒),不传表示不限。预算耗尽后不再抓新页, 已抓到的页面照常返回;排队中的子任务直接返回 timeout。 bypass_cache: 跳过并绕过响应缓存。 session: 登录态会话名(默认 default);对本次所有页面生效。 proxy: 本次爬取所有页面共用的代理;不传则按代理池分配,未配池时沿用全局配置。 显式传入代理时不走共享缓存。 region: 本次爬取所有页面共用的地区定向(如 us);只在走代理池时生效, 没有该地区出口时直接返回 invalid_argument。 ctx: MCP 上下文,由框架注入,用于逐页上报进度(客户端请求进度时才发通知)。

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "web_crawl"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYes
depthNo
limitNo
proxyNo
regionNo
cookiesNo
headersNo
sessionNodefault
timeoutNo
max_pagesNo
same_hostNo
max_tokensNo
deadline_msNo
link_policyNointernal
bypass_cacheNo
next_selectorNo

Output Schema

NameRequiredDescriptionDefault
resultYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv0.25.0
    • addedInput schema / properties / region
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "string"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Region"
      +}
  2. Addedv0.10.1

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are supplied, so the description carries the full burden and does so: it discloses the compliance pipeline (robots/allow-deny lists/rate limiting/same-host concurrency), per-page failure isolation ("单页失败只影响该页"), automatic skipping of static assets, and deadline semantics (queued subtasks return timeout while fetched pages are still returned). It also notes cache side effects (headers/cookies/explicit proxy bypass the shared cache) and the region failure mode returning invalid_argument. This is exactly the beyond-schema behavior an agent needs before a multi-page spend.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded summary sentence, then a short behavior paragraph, a two-bullet mode block that resolves the depth-vs-next_selector ambiguity, and a flat arg list. Nothing is repetitive against the schema and the structure matches how an agent would decide (mode first, then knobs). The single framework-injected ctx line is the only non-essential text.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 16-parameter crawling tool this is complete: entry conditions, both operating modes, per-page fault tolerance, budget/deadline behavior, cache and session interactions, and error cases are all covered. An output schema exists, so the omission of return-value details is appropriate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% across 16 parameters, so the description must compensate, and it does: every argument gets a one-line meaning plus non-obvious constraints (depth default 1 / max 3, limit bounded by crawl_max_pages, same_host default True, max_tokens per-page budget, deadline_ms as a total budget with partial-return semantics, session applying to all pages, region only effective on the proxy pool). Only link_policy defers to a sibling ("同 web_fetch") without defining the values.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with a concrete verb + resource: discover in-site links from a start page and batch-fetch them, or follow "next page" sequentially; the added "适合把一个栏目的前 N 篇读进来" gives a recognizable use shape. The two named modes (链接模式 / 分页模式) further pin down what the tool does. It stops short of naming which sibling it replaces (web_fetch for one page, web_batch for a known URL list).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states the mode-selection rule: pass next_selector (or rely on a site recipe) to enter pagination mode, and "有值时忽略 depth". Scope limits are given for both modes (depth max 3, limit capped by crawl_max_pages, max_pages default 5). Missing is an explicit when-not clause pointing at web_fetch/web_batch for single-page or fixed-list cases.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.