map
Discover every URL on a site in one call—scanning robots.txt, sitemaps, and homepage links. Get a de-duplicated list, total count, and section summary to size any site without full crawling.
Instructions
Discover every URL of a site in one call: robots.txt sitemaps + /sitemap.xml (including nested indexes) + same-domain homepage links, de-duplicated. Returns the list plus a site-wide total and a per-section summary. Always cheaper than crawling to find out how big a site is. $0.0005.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Seed URL. | |
| limit | No | Max URLs returned, default 100, cap 5000. Discovery always scans the whole site. | |
| search | No | Only return URLs containing this substring. | |
| country | No | ISO country code for the proxy exit. | |
| group_by | No | Set to "path" for the path tree with counts instead of the URL list. | |
| includeSubdomains | No | Include URLs on subdomains of the seed host. |