Skip to main content
Glama
scrape-mcp
by guanyuyan

web_index

Reads a site's sitemap, inventories page URLs, and compares them with the previous snapshot to report added, changed, unchanged, and removed pages, with optional regex filters and content diff checks.

Instructions

读取站点的 sitemap,清点页面 URL 并与上次的清单对比。

入口优先取 robots.txt 里的 Sitemap:,没有就回退 /sitemap.xml; sitemap index 会递归展开(最多 20 个文件、2 层),支持 .gz 压缩包。 清单存到 DATA_DIR/index/<host>.json,下次返回新增 / 变更 / 未变 / 移除的计数与 URL。

Args: url: 站点根或任意页面 URL(按 host 找 sitemap);也可直接给 sitemap 地址。 include: 只保留 URL 匹配该正则的条目;不传不过滤。 exclude: 排除 URL 匹配该正则的条目;不传不排除。 limit: 本次最多处理多少条 URL(默认 1000,硬上限 5000)。 check_content: True 时对 lastmod 变过的条目回抓正文并比哈希,内容真没变的不算 changed (单次最多回抓 20 个,避免把站点打穿)。 content_diff: True 时同样回抓正文,但额外与 web_watch 的快照对比并返回 changes(逐 URL 的增删行数与 diff 摘要);快照与 web_watch 共用, 下次单页 web_watch 可直接接续对比。dry_run 时不写快照。 dry_run: True 时只对比、不写入本次清单(便于先预览)。 reset: 丢弃旧清单并重建基线(本次返回 first_seen=true)。 session: 回抓正文时使用的登录态会话名(默认 default)。 deadline_ms: 整次调用的总预算(毫秒),不传表示不限;预算耗尽后不再展开新 sitemap。 proxy: 本次所有请求共用的代理;不传则按代理池分配,未配池时沿用全局配置。

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "web_index"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYes
limitNo
proxyNo
resetNo
dry_runNo
excludeNo
includeNo
sessionNodefault
deadline_msNo
content_diffNo
check_contentNo

Output Schema

NameRequiredDescriptionDefault
resultYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. Changed1 schema field changedv0.25.0
    • addedInput schema / properties / content_diff
      Added value: +{
      +  "default": false,
      +  "title": "Content Diff",
      +  "type": "boolean"
      +}
  2. Addedv0.21.0

TDQS

A4.3/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does so: sitemap discovery order (robots.txt Sitemap: then /sitemap.xml fallback), recursion caps (20 files, 2 levels), .gz support, the on-disk inventory path, the 20-refetch safeguard against hammering the site, dry_run's no-write semantics, reset discarding the old baseline, deadline exhausting sitemap expansion, and proxy-pool fallback behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

It is front-loaded with a one-line purpose followed by a clean Args block, and each entry carries real information. It is long, but the length is justified by the 11-parameter surface; only minor tightening (e.g. repeating '不传不过滤' style phrasing) would help.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For an 11-parameter, stateful, mutating-inventory tool with no annotations, the description covers discovery mechanics, recursion and rate safeguards, persistence location, dry-run/reset state semantics, and cross-tool snapshot sharing. An output schema exists, so the returned counts need not be formally specified, and the description still hints at first_seen/changes shape.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% across 11 parameters, so the description must compensate and it does: every argument is defined with meaning and non-obvious constraints — limit's hard ceiling of 5000, include/exclude as regex over URLs, check_content vs content_diff distinction and their 20-refetch cap, session default, deadline_ms budget semantics, and proxy pool fallback.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The first line states a specific verb+resource: read the site's sitemap, enumerate page URLs, and diff them against the previous inventory. It is clear what the tool does, but it never positions itself against the obvious siblings web_map (site URL mapping) and web_crawl, so the agent must infer the boundary.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied rather than stated: dry_run is described as a preview, reset as rebuilding a baseline, and content_diff is tied to the web_watch snapshot round-trip, which is genuinely useful routing info. However there is no explicit 'use this instead of web_map/web_crawl when…' guidance, and no prerequisites (auth, rate limits) are called out.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.