robots_sitemap
Passively read a site's robots.txt and the sitemap it points to. Returns the disallowed paths (often naming admin/staging areas — a recon signal), any declared crawl-delay, the sitemap URL(s), and — for the one sitemap fetched — whether it is a sitemap index, how many entries it holds, and a sample of them. Reads only these two public, owner-published files; it does NOT crawl, guess, or brute-force paths (that active content discovery is scan_site/Astur's job). Read-only.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| domain | Yes | Domain or URL, e.g. example.com |