Skip to main content
Glama

diogenes-web-scraping-engine-mcp

XML Sitemap & Index Crawler Tree Decompiler

crawl_sitemap_urls
Read-onlyIdempotent

Parses XML sitemaps and sitemap index files into structured URL trees with change frequency and last-modified filters. Use when discovering domain architectures, mapping crawl frontiers, or indexing blog/product catalogues; for extracting on-page hyperlinks from raw HTML, use extract_page_hyperlinks instead. Stateless read-only edge call (max 32 KB, 60 RPM): accepts 'sitemap_xml' ('filter_criteria' defaults to 'all_entries'), supports '?demo=true' or L402 (15 sats), and returns HTTP 400 if invalid.

Input Schema

NameRequiredDescriptionDefault
max_resultsNoMaximum number of URLs to extract and return in the catalog.
sitemap_xmlYesXML string of the sitemap or sitemapindex file to inspect and parse.
filter_criteriaNoURL retention filter: all entries, recently modified lastmod only, or HTML pages only.

Output Schema

NameRequiredDescriptionDefault
sub_sitemapsYesNested child sitemap URLs detected if document is a sitemapindex file.
sitemap_entriesYesList of structured URL entries containing loc, lastmod, and priority.
total_urls_foundYesTotal valid loc URLs parsed from the XML sitemap document.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the readOnly/idempotent/non-destructive annotations, it discloses payload cap (32 KB), rate limit (60 RPM), statelessness, demo-mode and L402 payment requirement (15 sats), and the 400 error on invalid input — none of which the annotations carry.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with purpose before routing guidance, and the operational facts are packed into a compact clause. Dense but essentially waste-free; only the parenthetical enumeration of limits feels slightly crammed.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With a 100%-covered schema, an output schema handling return values, and annotations covering the safety profile, the description still supplies limits, auth options, and failure mode. Nothing needed to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3; the description adds a real detail the schema lacks — that filter_criteria defaults to 'all_entries' — and names sitemap_xml as the accepted input. It does not, however, explain max_results semantics beyond the schema's own text.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Parses XML sitemaps and sitemap index files into structured URL trees') plus the filter dimension, and explicitly distinguishes itself from extract_page_hyperlinks. An agent can route between the two without opening either schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives concrete when-to-use scenarios (discovering domain architectures, mapping crawl frontiers, indexing blog/product catalogues) and names the alternative for the adjacent case, plus the condition that selects it ('raw HTML').

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources