map
Map a website's structure by discovering URLs from robots.txt, sitemaps, and homepage links, with summary counts and filtering.
Instructions
Discover a site's URLs fast (robots.txt sitemaps + /sitemap.xml + homepage links) without a full crawl. Returns up to limit URLs (default 100) plus the site-wide total and a per-section summary (e.g. '/blog': 1988) so you see the site's shape without the full list. Narrow with search (substring filter — the primary way to find specific pages) or set group_by 'path' for the path tree with counts instead of URLs.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The site URL to map | |
| limit | No | Max URLs returned (default 100). `total`/`summary` always cover the whole site. | |
| search | No | Only return URLs containing this substring — use this to narrow before raising limit | |
| group_by | No | path: return the path tree with per-prefix counts instead of the flat URL list | |
| includeSubdomains | No | Include subdomains of the seed host |