web_map
Map a site's URLs from robots.txt, sitemaps, feeds and homepage links without fetching pages. Use it before crawling to decide what to fetch and find topic pages with fewer tokens.
Instructions
List a site's URLs without fetching its pages: robots.txt, sitemaps (nested indexes and .gz), RSS/Atom feeds and homepage links. Use before web_crawl to decide what is worth fetching, or to find the page for a topic: a few hundred tokens instead of the thousands a crawl costs. Returns urls with sources_used, sitemaps and feeds; read the ones that matter with web_fetch_many.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Any URL on the site; the whole origin is mapped. | |
| limit | No | Maximum URLs to return. Default 1000. | |
| include | No | Only URLs containing this substring. | |
| sources | No | Sources to use: robots, sitemap, feeds, links. Default: all of them. |