Map a site
pagesift_mapList website URLs from sitemaps, robots.txt, and homepage links without fetching pages. Use this to find the right page before scraping or plan a batch scrape.
Instructions
List the URLs of a website from its sitemaps, robots.txt and homepage links, without fetching each page. Use it to find the right page before scraping, or to plan a batch scrape.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The site to map. | |
| limit | No | ||
| search | No | Only return URLs related to this term. | |
| sitemap | No | ||
| includeSubdomains | No | ||
| ignoreQueryParameters | No |