site_map
Discover up to 1,000 same-site URLs via robots.txt, sitemaps, and homepage links without fetching page content. Use it to decide which pages to read or crawl.
Instructions
Site Map (URL discovery) — List the URLs of a website without reading them: robots.txt, sitemaps and sitemap indexes plus homepage links, up to 1,000 same-site URLs with lastmod and source. A cheap first step to decide which pages to read with web-read-batch or site-crawl. Returns no page content. Params: url (site or domain, required), max_urls (1 to 1000, default 200). Cost: $0.0025 USDC per call. Latency p99: 8000ms.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Site URL or domain | |
| max_urls | No |