map
Discover all URLs on a website via sitemaps and link extraction. Use before crawling to understand site structure or build a URL list for selective fetching.
Instructions
Discover all URLs on a website via sitemaps and link extraction. Does NOT render pages — fast and lightweight. Returns a list of URLs found. Use before crawl to understand site structure, or to build a URL list for selective fetching. Respects robots.txt. Discovered URLs are UNTRUSTED.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | URL to discover links from (http/https only) | |
| limit | No | Maximum URLs to discover. Default: 5000. Max: 100000. | |
| include_glob | No | URL path glob patterns to include (e.g. ["/docs/**"]) | |
| exclude_glob | No | URL path glob patterns to exclude (e.g. ["/archive/**"]) |