scrape-mcp

Overview Schema Source Related Servers Discussions Admin

web_extract

Extract structured JSON from web pages by defining field schemas, returning targeted field values instead of full-page text for direct storage or analysis.

Instructions

抓取网页并按字段 schema 抽取结构化 JSON（字段级数据，可入库）。

适合需要"数据而非整页正文"的场景：给 URL 和字段规则，直接返回字段值，而不是一段 compact 文本。抓取链路（分级反爬、登录态、合规）与 web_fetch 一致。

字段规则示例： {"fields": { "title": {"selector": "h1"}, "price": {"selector": ".price", "regex": "(\d+(?:.\d+)?)", "group": 1}, "main_link": {"selector": "a", "type": "attr", "attr": "href", "abs_url": true}, "link_count": {"selector": "a", "type": "count"}, "tags": {"selector": ".tag", "type": "list", "list_key": "text"}, "items": {"selector": ".item", "type": "list", "fields": { "name": {"selector": ".name"}, "url": {"selector": "a", "type": "attr", "attr": "href", "abs_url": true} }}, "author": {"selector": ".author", "type": "object", "fields": { "name": {"selector": ".name"}, "bio": {"selector": ".bio"} }} }} 字段类型：text（默认，节点归一化文本）/ html / attr（需配 attr）/ count（匹配节点数）/ list（配 fields 返回对象数组，否则按 list_key 取值）/ object（在第一个匹配节点内部继续按 fields 抽取）/ jsonld（不写 selector，用 path 从 JSON-LD 取值，如 {"type": "jsonld", "path": "author.name"}）。后处理：regex + group（正则提取，未命中回落 default）、abs_url（相对链接转绝对）、 strip（去首尾空白，默认 true）、convert（int / float / date，date 输出 ISO 8601）、 required（必填，缺失时整体 ok=false 并返回 missing_required）。

Args: url: 目标网页完整 URL。 schema: 字段抽取规则字典，见上方示例；不传则套用该站 recipe 的 extract 规则。 headers: 自定义请求头；传入时不走共享缓存。 cookies: 自定义 cookie 字典；传入时不走共享缓存。 timeout: 单次请求超时秒数。 deadline_ms: 整次调用的总预算（毫秒），不传表示不限；降级规则同 web_fetch。 wait_for: 仅 L2 生效，等待该 CSS 选择器出现后再取内容；不传则套用站点 recipe。 scroll: 仅 L2 生效，滚动页面触发懒加载；不传则套用站点 recipe。 click: 仅 L2 生效，渲染后点击一次该 CSS 选择器；不传则套用站点 recipe。 bypass_cache: 跳过并绕过响应缓存，强制走网络。 debug: 返回 diagnostics（分层决策与 L2 交互命中情况），不含页面内容。 session: 登录态会话名（默认 default）；需与 login 时使用的会话名一致。 proxy: 本次请求使用的代理（如 http://127.0.0.1:7890）；不传则按代理池分配，未配池时沿用全局配置。显式传入代理的请求不读也不写共享缓存。

Input Schema

Name	Required	Default
`url`	Yes
`click`	No
`debug`	No
`proxy`	No
`schema`	No
`scroll`	No
`cookies`	No
`headers`	No
`session`	No	default
`timeout`	No
`wait_for`	No
`deadline_ms`	No
`bypass_cache`	No

Output Schema

Name	Required	Description	Default
`result`	Yes

Schema Changelog

Changes observed during successful MCP inspections.

Changed16 schema fields changedv0.10.1Oct 23, 2026

addedInput schema / properties / bypass_cache

Added value: +{
+  "default": false,
+  "title": "Bypass Cache",
+  "type": "boolean"
+}

addedInput schema / properties / click

Added value: +{
+  "anyOf": [
+    {
+      "type": "string"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null,
+  "title": "Click"
+}

addedInput schema / properties / cookies

Added value: +{
+  "anyOf": [
+    {
+      "additionalProperties": {
+        "type": "string"
+      },
+      "type": "object"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null,
+  "title": "Cookies"
+}

addedInput schema / properties / deadline_ms

Added value: +{
+  "anyOf": [
+    {
+      "type": "integer"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null,
+  "title": "Deadline Ms"
+}

addedInput schema / properties / debug

Added value: +{
+  "default": false,
+  "title": "Debug",
+  "type": "boolean"
+}

addedInput schema / properties / headers

Added value: +{
+  "anyOf": [
+    {
+      "additionalProperties": {
+        "type": "string"
+      },
+      "type": "object"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null,
+  "title": "Headers"
+}

addedInput schema / properties / proxy

Added value: +{
+  "anyOf": [
+    {
+      "type": "string"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null,
+  "title": "Proxy"
+}

removedInput schema / properties / schema / additionalProperties
```
Removed value: -true
```

addedInput schema / properties / schema / anyOf

Added value: +[
+  {
+    "additionalProperties": true,
+    "type": "object"
+  },
+  {
+    "type": "null"
+  }
+]

addedInput schema / properties / schema / default
```
Added value: +null
```
removedInput schema / properties / schema / type
```
Removed value: -"object"
```

addedInput schema / properties / scroll

Added value: +{
+  "anyOf": [
+    {
+      "type": "boolean"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null,
+  "title": "Scroll"
+}

addedInput schema / properties / session

Added value: +{
+  "default": "default",
+  "title": "Session",
+  "type": "string"
+}

addedInput schema / properties / timeout

Added value: +{
+  "anyOf": [
+    {
+      "type": "number"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null,
+  "title": "Timeout"
+}

addedInput schema / properties / wait_for

Added value: +{
+  "anyOf": [
+    {
+      "type": "string"
+    },
+    {
+      "type": "null"
+    }
+  ],
+  "default": null,
+  "title": "Wait For"
+}

changedInput schema / required

Previous value: -[
-  "url",
-  "schema"
-]New value: +[
+  "url"
+]

Addedv0.2.0Sep 18, 2026

TDQS

A4.7/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does so: it discloses cache-bypass semantics for headers/cookies/proxy, that wait_for/scroll/click only apply at L2, that debug returns diagnostics without page content, that session must match the login session name, that a missing required field yields ok=false with missing_required, and that an omitted schema falls back to the site recipe.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose and the web_fetch contrast are front-loaded, followed by the DSL example and an arg list. It is long but the length is justified by a 13-param DSL tool; the only slack is some repetition of the fetch-chain equivalence and a very large example block.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return-shape explanation is not required, yet the description still adds the failure contract (ok=false, missing_required). Combined with the full param semantics and DSL specification, nothing an agent needs to call this correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must compensate for all 13 parameters – and it does, documenting each arg's meaning, defaults (session=default), and interactions (proxy allocation from pool, deadline_ms degradation rules same as web_fetch). The field-rule DSL is fully specified with types, post-processing options, and worked examples.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource ('抓取网页并按字段 schema 抽取结构化 JSON') and immediately draws the distinction from sibling web_fetch ('数据而非整页正文', '而不是一段 compact 文本'). An agent can pick this over web_fetch without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It provides a clear use-case condition – when you need fields rather than full page text – and notes that the fetch chain (anti-bot tiers, login state, compliance) is identical to web_fetch. It does not explicitly say when to prefer web_fetch or web_crawl instead, so the routing guidance is implied rather than spelled out.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server