ContextFetch site map
contextfetch_mapList a website's URLs from sitemaps and start page links without fetching pages, to identify pages for extraction or crawling.
Instructions
List a website's URLs without fetching its pages: its sitemaps (from robots.txt, else /sitemap.xml) plus the start page's links, same site only, up to limit (<= 5,000). Use it to find the right pages before contextfetch_extract_url or contextfetch_crawl. Costs 1 credit.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Public http(s) URL of the site (any page on it). | |
| limit | No | Maximum URLs (1-5,000, default 100). | |
| exclude_paths | No | Regexes; never list URLs whose path matches one. | |
| include_paths | No | Regexes; only list URLs whose path matches one, e.g. ["^/blog/"]. |