site_map
Discover a site's real URLs and URL patterns from robots.txt sitemaps, root, or a polite crawl; filter by regex, refresh cached results, or use the Wayback index to avoid requests.
Instructions
A site's real urls (robots.txt sitemaps, a polite link walk, the catalogue's root) with their url patterns.
filter is a regex. A listing is reused for 12 hours; fresh=true lists the site again (only when it changed:
re-listing a whole site run after run is how crawlers get blocked). archive='only' lists the site from the Wayback
Machine's index without a single request to it; 'auto' does so when the site is on hold or listed nothing.
Use it instead of guessing paths.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| fresh | No | ||
| limit | No | ||
| filter | No | ||
| archive | No | auto |