Skip to main content
Glama
scrape-mcp
by guanyuyan

web_batch

Fetch multiple URLs concurrently through the full fetch pipeline, returning per-URL results with shared timeout, session, proxy, and format settings.

Instructions

并发抓取一批 URL,各自走完整分级链路,返回逐个结果。

Args: urls: 目标 URL 列表(最多 batch_max_urls 个,默认 20)。 max_tokens: 每个 URL 的正文 token 预算上限。 link_policy: 同 web_fetch。 format: 同 web_fetch(markdown / text / json / chunks)。 chunk_size: format=chunks 时的窗口字符数;对所有 URL 生效。 chunk_overlap: format=chunks 时相邻块的重叠字符数;对所有 URL 生效。 headers: 所有 URL 共用的自定义请求头;传入时不走共享缓存。 cookies: 所有 URL 共用的自定义 cookie;传入时不走共享缓存。 timeout: 单次请求超时秒数。 deadline_ms: 整批共用的总预算(毫秒),不传表示不限。预算耗尽后还没开始的 子任务直接返回 timeout(diagnostics 为 deadline:exceeded),不再占用网络与浏览器。 bypass_cache: 跳过并绕过响应缓存。 debug: 每条结果附带 diagnostics(决策轨迹与计数,不含页面内容)。 session: 登录态会话名(默认 default);对该批所有 URL 生效。 proxy: 该批所有 URL 共用的代理;不传则按代理池分配,未配池时沿用全局配置。 显式传入代理时该批不读也不写共享缓存。 region: 该批所有 URL 共用的地区定向(如 us);只在走代理池时生效, 没有该地区出口时直接返回 invalid_argument。 ctx: MCP 上下文,由框架注入,用于逐条上报进度(客户端请求进度时才发通知)。

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "urls": "<urls>"
    },
    "name": "web_batch"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlsYes
debugNo
proxyNo
formatNomarkdown
regionNo
cookiesNo
headersNo
sessionNodefault
timeoutNo
chunk_sizeNo
max_tokensNo
deadline_msNo
link_policyNointernal
bypass_cacheNo
chunk_overlapNo

Output Schema

NameRequiredDescriptionDefault
resultYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed3 schema fields changedv0.25.0
    • addedInput schema / properties / chunk_overlap
      Added value: +{
      +  "default": 200,
      +  "title": "Chunk Overlap",
      +  "type": "integer"
      +}
    • addedInput schema / properties / chunk_size
      Added value: +{
      +  "default": 1200,
      +  "title": "Chunk Size",
      +  "type": "integer"
      +}
    • addedInput schema / properties / region
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "string"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Region"
      +}
  2. Changed9 schema fields changedv0.10.1
    • addedInput schema / properties / bypass_cache
      Added value: +{
      +  "default": false,
      +  "title": "Bypass Cache",
      +  "type": "boolean"
      +}
    • addedInput schema / properties / cookies
      Added value: +{
      +  "anyOf": [
      +    {
      +      "additionalProperties": {
      +        "type": "string"
      +      },
      +      "type": "object"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Cookies"
      +}
    • addedInput schema / properties / deadline_ms
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "integer"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Deadline Ms"
      +}
    • addedInput schema / properties / debug
      Added value: +{
      +  "default": false,
      +  "title": "Debug",
      +  "type": "boolean"
      +}
    • addedInput schema / properties / format
      Added value: +{
      +  "default": "markdown",
      +  "title": "Format",
      +  "type": "string"
      +}
    • addedInput schema / properties / headers
      Added value: +{
      +  "anyOf": [
      +    {
      +      "additionalProperties": {
      +        "type": "string"
      +      },
      +      "type": "object"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Headers"
      +}
    • addedInput schema / properties / proxy
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "string"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Proxy"
      +}
    • addedInput schema / properties / session
      Added value: +{
      +  "default": "default",
      +  "title": "Session",
      +  "type": "string"
      +}
    • addedInput schema / properties / timeout
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "number"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Timeout"
      +}
  3. First observedv0.1.0

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations at all, the description carries the full burden and does substantial work: it explains that a request passing headers, cookies, or an explicit proxy bypasses the shared cache in both directions, that region only applies when a proxy pool is used and otherwise fails invalid_argument, and that the shared deadline causes not-yet-started subtasks to return timeout with diagnostics deadline:exceeded without touching network or browser. Remaining gaps are concurrency limits and the exact meaning of batch_max_urls.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose sentence is front-loaded and the Args block is dense but earns its space given 15 undocumented schema parameters. Some phrasing is more verbose than needed (the multi-line deadline and proxy explanations), but nothing is filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 15-parameter batch tool with no annotations and an output schema that already covers return values, the description supplies most of what an agent needs: per-URL vs whole-batch scope is clear for max_tokens, deadline_ms, headers, cookies, proxy, region, and session. The unresolved pieces are the batch_max_urls value and any notion of concurrency or ordering of results.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must document all 15 parameters, and it largely does — units are given for chunk_size/chunk_overlap (characters), timeout (seconds), deadline_ms (milliseconds), and max_tokens is scoped per URL rather than per batch, which prevents a real mistake. The weak spot is link_policy, documented only as '同 web_fetch' with no values or meaning, and format's enumeration is listed but not explained.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening line states a specific verb and resource — concurrent fetching of a batch of URLs, each through the full tiered pipeline, with per-URL results returned. This distinguishes it from the singular web_fetch, though the differentiation is carried by the word 'batch' rather than an explicit contrast statement.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied: batching multiple URLs is obviously the point, and several parameters are defined by reference to web_fetch ('同 web_fetch'). There is no explicit statement of when to prefer this over N separate web_fetch calls, nor any when-not guidance or sibling routing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.