map_website
Discover a website's page inventory and structure using sitemaps, robots.txt, and link graphs before crawling. Returns URLs with source and modification metadata.
Instructions
Discover the architecture and page inventory of a website domain via sitemaps, robots.txt, and link graphs. Returns a list of URLs with source and modification metadata. Useful for exploring a site before targeted crawling.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Target website URL or domain root (e.g. 'https://example.com'). | |
| max_links | No | Maximum number of links to discover (default: 100). |