Skip to main content
Glama
scrape-mcp
by guanyuyan

web_extract

Extract field-level structured JSON from web pages using URL and schema rules, returning specific data values instead of full-page text for storage or analysis.

Instructions

抓取网页并按字段 schema 抽取结构化 JSON(字段级数据,可入库)。

适合需要"数据而非整页正文"的场景:给 URL 和字段规则,直接返回字段值, 而不是一段 compact 文本。抓取链路(分级反爬、登录态、合规)与 web_fetch 一致。

字段规则示例: {"fields": { "title": {"selector": "h1"}, "price": {"selector": ".price", "regex": "(\d+(?:.\d+)?)", "group": 1}, "main_link": {"selector": "a", "type": "attr", "attr": "href", "abs_url": true}, "link_count": {"selector": "a", "type": "count"}, "tags": {"selector": ".tag", "type": "list", "list_key": "text"}, "items": {"selector": ".item", "type": "list", "fields": { "name": {"selector": ".name"}, "url": {"selector": "a", "type": "attr", "attr": "href", "abs_url": true} }}, "author": {"selector": ".author", "type": "object", "fields": { "name": {"selector": ".name"}, "bio": {"selector": ".bio"} }} }} 字段类型:text(默认,节点归一化文本)/ html / attr(需配 attr)/ count(匹配节点数)/ list(配 fields 返回对象数组,否则按 list_key 取值)/ object(在第一个匹配节点内部继续按 fields 抽取)/ jsonld(不写 selector,用 path 从 JSON-LD 取值,如 {"type": "jsonld", "path": "author.name"})。 后处理:regex + group(正则提取,未命中回落 default)、abs_url(相对链接转绝对)、 strip(去首尾空白,默认 true)、convert(int / float / date,date 输出 ISO 8601)、 required(必填,缺失时整体 ok=false 并返回 missing_required)。

Args: url: 目标网页完整 URL。 schema: 字段抽取规则字典,见上方示例;不传则套用该站 recipe 的 extract 规则。 prompt: 可选的自然语言抽取要求。配置了 LLM(SCRAPE_MCP_LLM_BASE_URL / MODEL)时 交给模型抽取;未配置或调用失败时自动回退确定性 schema,不产生默认外部调用。 字段规则里可用 description 说明字段含义,会一并传给模型。 grounding: 是否返回字段级溯源(默认取配置 grounding_default,关闭时载荷不变)。 开启后 data 的每个路径都会带 selector / snippet / offset / basis, 必填字段缺失时附 ≤3 个近似命中候选;LLM 抽取不提供溯源并写入 notes。 headers: 自定义请求头;传入时不走共享缓存。 cookies: 自定义 cookie 字典;传入时不走共享缓存。 timeout: 单次请求超时秒数。 deadline_ms: 整次调用的总预算(毫秒),不传表示不限;降级规则同 web_fetch。 wait_for: 仅 L2 生效,等待该 CSS 选择器出现后再取内容;不传则套用站点 recipe。 scroll: 仅 L2 生效,滚动页面触发懒加载;不传则套用站点 recipe。 click: 仅 L2 生效,渲染后点击一次该 CSS 选择器;不传则套用站点 recipe。 bypass_cache: 跳过并绕过响应缓存,强制走网络。 debug: 返回 diagnostics(分层决策与 L2 交互命中情况),不含页面内容。 session: 登录态会话名(默认 default);需与 login 时使用的会话名一致。 proxy: 本次请求使用的代理(如 http://127.0.0.1:7890);不传则按代理池分配, 未配池时沿用全局配置。显式传入代理的请求不读也不写共享缓存。 region: 地区定向(如 us / hk / jp),只在走代理池时生效;没有该地区出口时 直接返回 invalid_argument。会话绑定代理或显式 proxy 优先于它。

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "web_extract"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYes
clickNo
debugNo
proxyNo
promptNo
regionNo
schemaNo
scrollNo
cookiesNo
headersNo
sessionNodefault
timeoutNo
wait_forNo
groundingNo
deadline_msNo
bypass_cacheNo

Output Schema

NameRequiredDescriptionDefault
resultYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv0.35.0
    • addedInput schema / properties / grounding
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "boolean"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Grounding"
      +}
  2. Changed2 schema fields changedv0.25.0
    • addedInput schema / properties / prompt
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "string"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Prompt"
      +}
    • addedInput schema / properties / region
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "string"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Region"
      +}
  3. Changed16 schema fields changedv0.10.1
    • addedInput schema / properties / bypass_cache
      Added value: +{
      +  "default": false,
      +  "title": "Bypass Cache",
      +  "type": "boolean"
      +}
    • addedInput schema / properties / click
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "string"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Click"
      +}
    • addedInput schema / properties / cookies
      Added value: +{
      +  "anyOf": [
      +    {
      +      "additionalProperties": {
      +        "type": "string"
      +      },
      +      "type": "object"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Cookies"
      +}
    • addedInput schema / properties / deadline_ms
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "integer"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Deadline Ms"
      +}
    • addedInput schema / properties / debug
      Added value: +{
      +  "default": false,
      +  "title": "Debug",
      +  "type": "boolean"
      +}
    • addedInput schema / properties / headers
      Added value: +{
      +  "anyOf": [
      +    {
      +      "additionalProperties": {
      +        "type": "string"
      +      },
      +      "type": "object"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Headers"
      +}
    • addedInput schema / properties / proxy
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "string"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Proxy"
      +}
    • removedInput schema / properties / schema / additionalProperties
      Removed value: -true
    • addedInput schema / properties / schema / anyOf
      Added value: +[
      +  {
      +    "additionalProperties": true,
      +    "type": "object"
      +  },
      +  {
      +    "type": "null"
      +  }
      +]
    • addedInput schema / properties / schema / default
      Added value: +null
    • removedInput schema / properties / schema / type
      Removed value: -"object"
    • addedInput schema / properties / scroll
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "boolean"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Scroll"
      +}
    • addedInput schema / properties / session
      Added value: +{
      +  "default": "default",
      +  "title": "Session",
      +  "type": "string"
      +}
    • addedInput schema / properties / timeout
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "number"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Timeout"
      +}
    • addedInput schema / properties / wait_for
      Added value: +{
      +  "anyOf": [
      +    {
      +      "type": "string"
      +    },
      +    {
      +      "type": "null"
      +    }
      +  ],
      +  "default": null,
      +  "title": "Wait For"
      +}
    • changedInput schema / required
      Previous value: -[
      -  "url",
      -  "schema"
      -]New value: +[
      +  "url"
      +]
  4. Addedv0.2.0

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does so: the tiered anti-bot/login/compliance pipeline, cache behavior (headers, cookies and explicit proxy all bypass the shared cache), LLM extraction being off by default with deterministic fallback, grounding driven by config, deadline degradation, and region returning invalid_argument when no exit exists. Failure semantics (ok=false plus missing_required, notes for LLM runs) are also disclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Purpose is front-loaded in the first sentence, followed by behavior, then a schema example and per-argument notes — a sensible order. It is long, and the example block is large, but that length is largely earned by a 16-parameter free-form DSL; a few lines (e.g. restating the example in the schema arg entry) are mild repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex, 16-parameter extraction tool, the description supplies everything an agent needs: when to choose it, how the field DSL works, fallback paths, cache and auth preconditions, and error signals. An output schema exists, and the description correctly does not waste space re-describing the return payload beyond the ok/missing_required contract.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must compensate, and it documents all 16 parameters individually — including defaults and precedence rules (session vs explicit proxy vs region, wait_for/scroll/click only applying at L2, schema falling back to the site recipe). The field-rule DSL, which the schema types only as a free-form object, is fully specified with type, post-processing and required-field error behavior.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening line gives a specific verb and resource — fetch a page and extract structured JSON by field schema — and immediately contrasts it with web_fetch ("field-level data, not a compact text block"). An agent can distinguish the two without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It states the selecting condition clearly ("scenarios needing data rather than full page text") and notes the fetch pipeline matches web_fetch, which implicitly tells the agent which tool to pick. It stops short of an explicit "do not use this when…" rule or naming other siblings such as web_crawl/web_batch for multi-page work.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.