Skip to main content
Glama
scrape-mcp
by guanyuyan

web_fetch

Fetches web pages and returns clean, token-efficient article text while handling anti-bot blocks, login walls, and JavaScript rendering for reliable content extraction.

Instructions

抓取网页并返回极省 token 的紧凑正文(自动辨别反爬拦截页并如实上报)。

支持 HTML 页面、PDF(需安装 mcp-scrape[pdf];扫描件可开 OCR)、 DOCX/XLSX/PPTX(标准库抽取,无需额外依赖)与纯文本(.txt/.md); 非 HTML 内容直接返回文本,不做 DOM 精简。

Args: url: 目标网页完整 URL,需带 http/https 协议头。 max_tokens: 正文 token 预算上限,超出时保留头尾、省略中段。 link_policy: 链接处理策略。internal=站内链接保留为相对路径、站外降级为纯文本(默认,最省); all=站内外链接全保留;none=全部降级为纯文本。 format: 输出格式。markdown=保留标题/列表/链接标记(默认); text=剥掉 markdown 标记的纯文本;json=按行切分的块数组(content 为字符串列表); chunks=RAG 分块数组(固定字符窗口 + overlap + heading_path,可直接入库)。 chunk_size: format=chunks 时的窗口字符数(默认 1200,范围 1008000)。 chunk_overlap: format=chunks 时相邻块的重叠字符数(默认 200,范围 0chunk_size/2)。 headers: 自定义请求头,如 {"Referer": "...", "User-Agent": "..."};传入时不走共享缓存。 cookies: 自定义 cookie 字典;传入时不走共享缓存。 timeout: 单次请求超时秒数(默认取配置 request_timeout)。 deadline_ms: 整次调用的总预算(毫秒),不传表示不限。预算不够时逐级降级: 压缩单步超时 → 跳过重试(diagnostics 里是 deadline:skip_retry)→ 不升级 L2(deadline:skip_l2)→ 未开始就耗尽时直接返回 timeout。 预算内拿到的正文照常返回,但降级结果不会写入结果缓存。 wait_for: 仅 L2 生效,等待该 CSS 选择器出现后再取内容(适合 SPA);不传则套用站点 recipe。 scroll: 仅 L2 生效,滚动页面触发懒加载/无限滚动;不传则套用站点 recipe。 click: 仅 L2 生效,渲染后点击一次该 CSS 选择器(如"加载更多");不传则套用站点 recipe。 actions: 仅 L2 生效,按顺序执行动作链(click/type/press/select/wait/scroll), 适合"输入关键词 → 点击搜索 → 等待结果"这类交互;单个动作失败只写入 notes, 后续动作继续。传 actions 时会跳过 L1 直接渲染,浏览器不可用时如实提示动作未执行。 bypass_cache: 跳过并绕过响应缓存,强制走网络。 debug: 返回 diagnostics(分层决策、L2 交互命中情况、正文根与候选命中数), 只含计数与决策、不含页面内容;失败时默认也会带精简版决策轨迹。 session: 登录态会话名(默认 default);需与 login 时使用的会话名一致,多身份互不干扰。 proxy: 本次请求使用的代理(如 http://127.0.0.1:7890、socks5://127.0.0.1:1080), L1 与升级后的 L2 都走该出口;不传则按代理池 SCRAPE_MCP_PROXY_POOL 分配, 未配池时沿用 SCRAPE_MCP_PROXY。 显式传入代理的请求不读也不写共享缓存,避免把不同出口的结果混在一起。 region: 地区定向(如 us / hk / jp),只在走代理池时生效:仅从带该地区标签的 出口中选择。池里没有该地区时直接返回 invalid_argument,不会悄悄换出口。 会话已在 SCRAPE_MCP_PROXY_SESSIONS 绑定代理、或显式传了 proxy 时不生效。 query: 传入时只返回与问题最相关的段落(结构同 web_search(read="excerpt")): content 换成命中段落、附 excerpts/offset,召回不足时回退整页并标记 excerpt_fallback。仅支持 format=markdown/text。 version: 读取历史快照而不发请求:latest(默认最新)/ prev(上一版)/ 序号 (1=最新)/ ISO 时间点。用于"上周那版长什么样"这类回溯,不抓实时页面; 没有快照时返回 no_snapshot。

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "web_fetch"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYes
clickNo
debugNo
proxyNo
queryNo
formatNomarkdown
regionNo
scrollNo
actionsNo
cookiesNo
headersNo
sessionNodefault
timeoutNo
versionNo
wait_forNo
chunk_sizeNo
max_tokensNo
deadline_msNo
link_policyNointernal
bypass_cacheNo
chunk_overlapNo

Output Schema

NameRequiredDescriptionDefault
resultYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed2 schema fields changedv0.35.0
    • addedInput schema / properties / query
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "string"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Query"
      +}
    • addedInput schema / properties / version
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "string"
      +    },
      +    {
      +      "type": "integer"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Version"
      +}
  2. Changed3 schema fields changedv0.25.0
    • addedInput schema / properties / chunk_overlap
      Added value: +{
      +  "default": 200,
      +  "title": "Chunk Overlap",
      +  "type": "integer"
      +}
    • addedInput schema / properties / chunk_size
      Added value: +{
      +  "default": 1200,
      +  "title": "Chunk Size",
      +  "type": "integer"
      +}
    • addedInput schema / properties / region
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "string"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Region"
      +}
  3. Changed1 schema field changedv0.21.0
    • addedInput schema / properties / actions
      Added value: +{
      +  "anyOf": [
      +    {
      +      "items": {
      +        "additionalProperties": true,
      +        "type": "object"
      +      },
      +      "type": "array"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Actions"
      +}
  4. Changed12 schema fields changedv0.10.1
    • addedInput schema / properties / bypass_cache
      Added value: +{
      +  "default": false,
      +  "title": "Bypass Cache",
      +  "type": "boolean"
      +}
    • addedInput schema / properties / click
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "string"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Click"
      +}
    • addedInput schema / properties / cookies
      Added value: +{
      +  "anyOf": [
      +    {
      +      "additionalProperties": {
      +        "type": "string"
      +      },
      +      "type": "object"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Cookies"
      +}
    • addedInput schema / properties / deadline_ms
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "integer"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Deadline Ms"
      +}
    • addedInput schema / properties / debug
      Added value: +{
      +  "default": false,
      +  "title": "Debug",
      +  "type": "boolean"
      +}
    • addedInput schema / properties / format
      Added value: +{
      +  "default": "markdown",
      +  "title": "Format",
      +  "type": "string"
      +}
    • addedInput schema / properties / headers
      Added value: +{
      +  "anyOf": [
      +    {
      +      "additionalProperties": {
      +        "type": "string"
      +      },
      +      "type": "object"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Headers"
      +}
    • addedInput schema / properties / proxy
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "string"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Proxy"
      +}
    • addedInput schema / properties / scroll
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "boolean"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Scroll"
      +}
    • addedInput schema / properties / session
      Added value: +{
      +  "default": "default",
      +  "title": "Session",
      +  "type": "string"
      +}
    • addedInput schema / properties / timeout
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "number"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Timeout"
      +}
    • addedInput schema / properties / wait_for
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "string"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Wait For"
      +}
  5. First observedv0.1.0

TDQS

A4.3/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full behavioral burden and does so in detail: cache read/write rules (headers, cookies, explicit proxy all bypass the shared cache), the deadline degradation ladder with named diagnostics states (deadline:skip_retry, deadline:skip_l2, timeout), L2-only semantics for wait_for/scroll/click/actions, and per-action failure notes. It also discloses auth-dependent behavior (session must match login), region selection returning invalid_argument rather than silently switching exits, and no_snapshot. This is unusually complete behavioral disclosure.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The summary sentence is front-loaded and the Args block is well organized, with each entry earning its place given 21 undocumented parameters. It is long, but the length is largely justified by the schema's 0% coverage; a few entries (e.g. the multi-line deadline degradation description) could be tightened.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return-value explanation is not required, and the description still notes key return shapes where it matters (diagnostics contain counts only, excerpts/offset, no_snapshot, excerpt_fallback). For a 21-parameter tool with no annotations, the coverage of caching, degradation, sessions, proxies, and region is sufficient for an agent to call it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must carry all 21 parameters — and it does, documenting each with defaults, ranges, and semantics (chunk_size 100–8000, chunk_overlap 0–chunk_size/2, link_policy internal/all/none, format markdown/text/json/chunks, proxy and region interaction rules). It adds meaning well beyond the bare schema, including cross-parameter constraints such as query only supporting format=markdown/text.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening sentence states a specific verb and resource: fetch a web page and return a token-thrifty compact body, with anti-scraping block pages detected and reported honestly. It also enumerates supported content types (HTML, PDF, DOCX/XLSX/PPTX, plain text) and the non-HTML behavior. What it lacks is an explicit statement of how this differs from siblings like web_extract or web_crawl, so it is clear but not sibling-differentiating.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied through parameter-level conditionals: version is described as the way to read a historical snapshot 'instead of fetching live', and query as returning only relevant passages. There is no explicit 'use this tool when…, use web_crawl/web_extract when…' routing, so an agent must infer when a single-URL fetch is the right choice.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.