Skip to main content
Glama
guanyuyan
by guanyuyan

web_index

Reads a site's sitemap, inventories page URLs, and compares them with the previous snapshot to report added, changed, unchanged, and removed entries.

Instructions

读取站点的 sitemap,清点页面 URL 并与上次的清单对比。

入口优先取 robots.txt 里的 Sitemap:,没有就回退 /sitemap.xml; sitemap index 会递归展开(最多 20 个文件、2 层),支持 .gz 压缩包。 清单存到 DATA_DIR/index/<host>.json,下次返回新增 / 变更 / 未变 / 移除的计数与 URL。

Args: url: 站点根或任意页面 URL(按 host 找 sitemap);也可直接给 sitemap 地址。 include: 只保留 URL 匹配该正则的条目;不传不过滤。 exclude: 排除 URL 匹配该正则的条目;不传不排除。 limit: 本次最多处理多少条 URL(默认 1000,硬上限 5000)。 check_content: True 时对 lastmod 变过的条目回抓正文并比哈希,内容真没变的不算 changed (单次最多回抓 20 个,避免把站点打穿)。 dry_run: True 时只对比、不写入本次清单(便于先预览)。 reset: 丢弃旧清单并重建基线(本次返回 first_seen=true)。 session: 回抓正文时使用的登录态会话名(默认 default)。 deadline_ms: 整次调用的总预算(毫秒),不传表示不限;预算耗尽后不再展开新 sitemap。 proxy: 本次所有请求共用的代理;不传则按代理池分配,未配池时沿用全局配置。

Input Schema

NameRequiredDescriptionDefault
urlYes
limitNo
proxyNo
resetNo
dry_runNo
excludeNo
includeNo
sessionNodefault
deadline_msNo
check_contentNo

Output Schema

NameRequiredDescriptionDefault
resultYes

Schema Changelog

Changes observed during successful MCP inspections.

  1. Addedv0.21.0

TDQS

A4.3/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations, the description carries the full burden and does so well: it discloses where state is persisted (DATA_DIR/index/<host>.json), recursion caps (20 files, 2 layers), the 20-URL refetch budget for content checks, the deadline_ms budget behavior, proxy-pool fallback, and that reset discards the old baseline. The dry_run flag is explained as a preview-only mode, and reset's destructive effect on the baseline is stated explicitly.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The purpose is front-loaded in the first line, followed by sitemap-discovery rules and then a compact per-parameter list. Every sentence adds information, but the block is long and the parameter documentation is dense enough that an agent must scan carefully; a short summary line before the Args would tighten it.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 10-parameter tool with no annotations and an output schema, the description covers discovery fallbacks, recursion limits, persistence path, diff categories (added/changed/unchanged/removed), and each parameter's behavior. The output schema handles return shape, so nothing essential is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must fully compensate, and the Args block documents all 10 parameters with semantics, defaults (limit 1000, hard cap 5000), and edge behavior (include/exclude regex, no filter when omitted; check_content only refetches lastmod-changed entries). This is richer than any schema would typically provide.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening line states a specific verb and resource: read the site's sitemap, enumerate page URLs, and diff against the previous list. The scope is concrete (sitemap discovery via robots.txt or /sitemap.xml, recursive expansion) and an agent can tell this is an inventory/diff tool. It doesn't explicitly say how it differs from sibling web_map, which likely also produces URL maps, so it stops short of 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the diff/baseline framing: use it to track which pages appeared, changed, or disappeared on a site. However there is no explicit when-to-use or when-not-to-use guidance, and no comparison against the 15 sibling tools (web_map, web_crawl, web_watch) that plausibly overlap. The reader must infer the routing decision.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.