crw_map
Discover all URLs on a website by reading its sitemap and optionally crawling to find pages not listed in the sitemap.
Instructions
Discover URLs on a website by crawling and/or reading its sitemap.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| crawlFallback | No | If true (default), supplements sitemap discovery with a short BFS crawl when the sitemap returns enough URLs. Set false for sitemap-only mode (faster, may miss pages not in the sitemap). | |
| maxDepth | No | Maximum crawl depth for discovery (default: 2) | |
| url | Yes | The URL to map | |
| useSitemap | No | Whether to use the site's sitemap.xml (default: true) |