Skip to main content
Glama

Discover URLs from a site's sitemaps

scrapyard_map
Read-only

Discover site URLs from robots.txt and sitemap XML without crawling. Get a clean list of published pages for documentation, blog, or product inventory before extraction.

Instructions

Discover a site's published URLs from robots.txt and sitemap XML without crawling pages.

Use this to inventory documentation, blog, or product URLs before selecting pages to extract. The response reports the discovery source, total count, filtered_count, and URL/lastmod entries. It intentionally does not follow ordinary page links or perform a full crawl.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
urlYesAbsolute public HTTP(S) site URL whose robots.txt and sitemap XML should be inspected.
limitNoMaximum number of safe sitemap URLs to return (1-5000).

Output Schema

TableJSON Schema
NameRequiredDescriptionDefault

No arguments

Install Server

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations include readOnlyHint=true and openWorldHint=true, so safety profile is already declared. Description adds that it intentionally does not follow links or crawl, and reports source, count, filtered_count, and entries. This adds behavioral context beyond annotations without contradicting them.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two compact paragraphs. First sentence states purpose and scope immediately. Second paragraph gives usage guidance and output details. No fluff, all sentences earn their place.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, the description need not detail return values, and it only briefly mentions the source/count fields. It effectively communicates the tool's non-crawling nature and inventory use case. Slight gap: no mention of rate limits or error behavior, but minor given annotations and schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% with descriptions for both parameters. The description mentions the discovery source and counts, and while it implies the output shape, it doesn't add parameter-level syntax beyond schema. Baseline 3 is appropriate when schema covers all parameters.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clear verb+resource: 'Discover a site's published URLs from robots.txt and sitemap XML'. Distinguishes from crawling. Sibling tools include scrapyard_extract and scrapyard_search, and description explicitly states it does not follow page links or crawl, so differentiation is clear.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

States when to use: 'inventory documentation, blog, or product URLs before selecting pages to extract'. Implies alternative scrapyard_extract but doesn't explicitly name alternatives. Provides context but no exclusions or direct comparison to siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Other Tools

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/artboarding-hash/scrapyard'

If you have feedback or need assistance with the MCP directory API, please join our Discord server