map_website
Discover all URLs from a website using sitemaps, CommonCrawl, or in-site crawling. Get a complete list for scraping, analysis, or site mapping.
Instructions
Discover URLs from a website using sitemaps, CommonCrawl, or in-site crawling.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| domain | Yes | ||
| sources | No | ||
| max_urls | No | ||
| async_mode | No | ||
| url_pattern | No | ||
| check_if_live | No |