Discover URLs from a site's sitemaps
scrapyard_mapDiscover site URLs from robots.txt and sitemap XML without crawling. Get a clean list of published pages for documentation, blog, or product inventory before extraction.
Instructions
Discover a site's published URLs from robots.txt and sitemap XML without crawling pages.
Use this to inventory documentation, blog, or product URLs before selecting pages to extract.
The response reports the discovery source, total count, filtered_count, and URL/lastmod
entries. It intentionally does not follow ordinary page links or perform a full crawl.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Absolute public HTTP(S) site URL whose robots.txt and sitemap XML should be inspected. | |
| limit | No | Maximum number of safe sitemap URLs to return (1-5000). |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||