Discover a site's URLs without fetching page content. Reads robots.txt sitemap directives, sitemap.xml and nested sitemap indexes, falling back to homepage link extraction. Price: $0.004
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to process | |
| limit | No | Maximum URLs to return | |
| exclude | No | Skip URLs whose path+query contains one of these | |
| include | No | Only URLs whose path+query contains one of these | |
| timeout | No | Request timeout in milliseconds |
Changes observed during successful MCP inspections.
Input schema / properties / exclude / defaultAdded value: +[]Input schema / properties / exclude / descriptionPrevious value: -"Skip URLs whose path+query contains one of these substrings"New value: +"Skip URLs whose path+query contains one of these"Input schema / properties / exclude / items / maxLengthAdded value: +200Input schema / properties / exclude / items / minLengthAdded value: +1Input schema / properties / exclude / maxItemsAdded value: +20Input schema / properties / include / defaultAdded value: +[]Input schema / properties / include / descriptionPrevious value: -"Only URLs whose path+query contains one of these substrings"New value: +"Only URLs whose path+query contains one of these"Input schema / properties / include / items / maxLengthAdded value: +200Input schema / properties / include / items / minLengthAdded value: +1Input schema / properties / include / maxItemsAdded value: +20Input schema / properties / limit / defaultAdded value: +1000Input schema / properties / limit / descriptionPrevious value: -"Maximum URLs to return (1-5000, default: 1000)"New value: +"Maximum URLs to return"Input schema / properties / limit / exampleAdded value: +1000Input schema / properties / limit / maximumAdded value: +5000Input schema / properties / limit / minimumAdded value: +1Input schema / properties / timeout / defaultAdded value: +10000Input schema / properties / timeout / descriptionPrevious value: -"Timeout in ms (default 10000)"New value: +"Request timeout in milliseconds"Input schema / properties / url / descriptionPrevious value: -"Site URL to map"New value: +"The URL to process"Input schema / properties / url / exampleAdded value: +"https://example.com"Input schema / properties / url / formatAdded value: +"uri"Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries the full burden, and it does disclose real mechanism: robots.txt sitemap directives, sitemap.xml, nested sitemap indexes, and homepage link-extraction fallback, plus a price. It omits failure behavior, rate limits, and whether results are deduplicated or sorted.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences plus a price tag; the core capability is front-loaded and the fallback chain is compressed into one clause with no filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
No output schema exists, but the return is self-evident from the name and the 'limit'/'Maximum URLs to return' parameter. Given full schema coverage and no annotations, the description covers what an agent needs to call it correctly, though it never states the response shape explicitly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 100%, so all five parameters (url, limit, exclude, include, timeout) are already documented in the schema. The description adds no syntax or filtering nuance beyond that, so baseline 3 applies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Specific verb+resource: 'Discover a site's URLs.' The phrase 'without fetching page content' implicitly distinguishes it from fetch/crawl siblings, which is genuinely useful differentiation. However, it never names crawl_site, the closest sibling, so an agent must infer the boundary itself.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Usage context is implied by 'without fetching page content' but there is no explicit when-to-use, when-not-to-use, or named alternative (e.g. crawl_site for full-page content). An agent can guess the intent but gets no routing guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Add one secure layer between your agents and this server.