List the URLs in a site's sitemaps
extract_sitemap_urlsExtract every URL from a website's sitemaps or a direct sitemap XML, including lastmod, changefreq, and priority. Filter by regex or recent changes to map pages before crawling.
Instructions
Give a site (sitemaps are discovered through robots.txt) or a direct sitemap.xml URL and get every listed URL with lastmod, changefreq and priority. Follows sitemap indexes and reads .gz files. Filter with includePatterns/excludePatterns (regular expressions) and lastmodAfter or lastmodWithinDays. Use it to map a site's pages or find recently changed ones before crawling. Cost, charged to your own Apify account: $0.0002 per URL returned. Each call is capped at maxCostUsd (default $0.25). Defaults applied by this server when omitted: maxTotalUrls=500. Runs Apify Actor fecklesspleb/sitemap-extractor (https://apify.com/fecklesspleb/sitemap-extractor).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| startUrls | No | Websites or sitemap URLs. A site (https://www.nasa.gov) to auto-discover its sitemaps, or a direct sitemap URL. Leave empty, or keep the sample NASA sitemap, for a default-mode run: up to 40 sample URLs from https://www.nasa.gov/wp-sitemap.xml at the normal per-URL price. | |
| maxCostUsd | No | Most this call may charge your Apify account, in US dollars. Default 0.25, maximum 5. Passed to Apify as maxTotalChargeUsd, the most Apify will charge for this run. | |
| onlyNewUrls | No | Only new URLs (monitoring). Return only URLs not seen in earlier runs with the same state store. | |
| lastmodAfter | No | Changed after (date). Keep URLs whose lastmod is on or after this date, YYYY-MM-DD. URLs with no lastmod are dropped unless you turn on Keep URLs without a lastmod. | |
| maxTotalUrls | No | Max URLs in total. 0 = no limit. | |
| maxUrlsPerSite | No | Max URLs per site. 0 = no limit. | |
| stateStoreName | No | State store name. Named key-value store for monitoring. Use a different name per monitor. | sitemap-extractor-state |
| excludePatterns | No | Exclude URL patterns (regex). Drop URLs matching any of these regular expressions. | |
| includePatterns | No | Include URL patterns (regex). Keep URLs matching any of these regular expressions, for example /blog/. | |
| keepUndatedUrls | No | Keep URLs without a lastmod. Off by default: when a date filter is set, URLs with no lastmod in the sitemap are dropped, since their age can't be checked. Turn this on to keep them. On sites whose sitemaps have no lastmod, that returns every URL (up to the limits), and each one is billed. | |
| sitemapPatterns | No | Only follow child sitemaps matching (regex). For big sitemap indexes, follow only child sitemaps matching these, for example post-sitemap. | |
| includeAlternates | No | Include hreflang alternates. Add language alternate URLs listed in the sitemap. | |
| lastmodWithinDays | No | Changed within last N days. Keep URLs whose lastmod is within the last N days. Overrides the date above. URLs with no lastmod are dropped unless you turn on Keep URLs without a lastmod. | |
| maxSitemapsPerSite | No | Max sitemap files per site. Safety cap on sitemap files read per site. |