Skip to main content
Glama

zato: Ethereum & Base Data API (pay-per-call x402)

robots.txt and sitemap.xml analysis: lists sitemap URLs and crawl rules for a site

web_sitemap
Read-only

Get sitemap URLs and robots.txt rules. Read a site's robots.txt (sitemaps, disallow rules for *) and its XML sitemap (or sitemap index, .xml.gz supported). Returns URL counts, child sitemaps and up to 500 URLs with lastmod. Price: $0.001 USDC per call (x402 v2, Base). Free demo: https://api.zato.org/demo/web/sitemap

Tool call template

Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {
      "url": "<url>"
    },
    "name": "web_sitemap"
  }
}

Input Schema

NameRequiredDescriptionDefault
urlYesAny URL of the site, or a direct sitemap URL
limitNoMax URLs to return

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A3.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and openWorldHint=true, so the safety profile and arbitrary-URL reach are covered structurally. The description adds genuinely useful context beyond that: it discloses the return shape (URL counts, child sitemaps, up to 500 URLs with lastmod), gzip sitemap support, and the $0.001 USDC x402 pricing, which an agent must know before calling. It stops short of noting failure modes such as a missing robots.txt/sitemap or robots disallow semantics.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three compact sentences, front-loaded with the action and its scope, then the return contents, then commercial terms. Dense but every sentence carries information; the pricing and demo clauses are long-ish but materially useful for an agent deciding whether to call.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description should explain returns, and it does (counts, child sitemaps, up to 500 URLs with lastmod). Only two simple parameters and both are documented. The main gap is edge-case behavior when robots.txt or the sitemap is absent or disallowed, which matters for an open-world fetcher.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so both parameters are already fully documented, and the description largely repeats rather than extends them: "any URL of the site, or a direct sitemap URL" is verbatim what the schema says for url. The "up to 500 URLs" phrasing restates the schema maximum. Baseline 3 applies because the schema carries the parameter burden.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and pair of resources ("Get sitemap URLs and robots.txt rules") and then expands on exactly what is read: sitemaps, disallow rules for *, XML sitemap or sitemap index, .gz support. None of the sibling tools (web_markdown, web_meta, domain_check) cover this combination, so an agent can distinguish it without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the content it returns (robots rules, sitemap URLs, lastmod), which tells an agent it is a crawl-discovery tool, but there is no explicit when-to-use or when-not-to-use statement and no named alternative among the sibling tools. The demo URL gives a way to trial it, which helps but is not guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources