Map a website
mapBuild a cached link inventory for a website by combining sitemap discovery with homepage link extraction, then filter and paginate internal, external, or subdomain URLs before scraping.
Instructions
Build (or fetch a cached) link-map for a website by combining sitemap discovery with homepage link extraction. Returns paginated lists of discovered URLs, each item just { url }. Filterable by link type (internal / external / subdomain). Use this to enumerate a site before scraping selected pages. max_urls (default 5000, max 100000) is a discovery budget, not a trim at the end: discovery stops as soon as that many URLs are found, so a smaller value is a faster, cheaper crawl. A map stopped that way returns exactly max_urls links with response_capped false — the signal that the site has more is response_meta.truncation.discovery_cap_reason. When that is "max_urls", ask again with a higher max_urls to get more; "unread_files" means a sitemap file could not be read this time and is often temporary, so asking again later can return more; "time", "file_budget", "depth" and "file_size" mean the site itself is big, slow or deep and a retry will not help. discovery_capped says discovery stopped early, sitemaps_skipped how many sitemap files were skipped or only partly read. limit (default 5000, max 10000) only pages the answer. Set zero_data_retention: true to keep the result out of the shared cache; anything stored to deliver it is kept for 24 hours, then deleted. It adds 1 credit and must be enabled for the organization (else a zero_data_retention_not_enabled error, not billed). See https://crawlbrulee.com/docs/zero-data-retention. Returned URLs are normalized the same way scrape normalizes its returned url, so map-then-scrape stays on one host. Results are ordered with the most useful links first. The response carries response_meta.usage: total_credit_cost = engine_credit_cost × proxy_multiplier + zero_data_retention_credit_cost, plus engine and proxy (the resolved tier, never auto). Map has no screenshot slicing and no page_status_code: it reads many files, not one page. A map that found nothing because the site answered only with statuses we don't bill (a 5xx, for example), or not at all, is empty and free. If the site cannot be reached, the tool can return a target_unreachable error (HTTP 502, not billed): retry later or check the URL.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The website URL to map. Mapping always targets the site root, so the path, query string and fragment are dropped; known tracking parameters are removed before the request is processed. | |
| page | No | Page number for paginated results | |
| cache | No | Cache settings for this request | |
| limit | No | Number of URLs to return per page. Default 5000, maximum 10000. | |
| proxy | No | Proxy tier to use for fetching | auto |
| types | No | Filter which link types to include | |
| location | No | Optional country emulation for the map | |
| max_urls | No | Maximum number of URLs to discover and store in the map. Default 5000, maximum 100000. Sitemap discovery stops as soon as this many URLs have been found, so a smaller value is a faster and lighter crawl, not just a smaller answer. | |
| sitemap_only | No | Only use sitemap.xml — skip homepage link extraction | |
| zero_data_retention | No | Keeps the result out of the shared cache; anything stored to deliver it is kept for 24 hours, then deleted. Adds 1 credit. Must be enabled for your organization. See https://crawlbrulee.com/docs/zero-data-retention. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| links | Yes | List of discovered URLs for the current page | |
| response_meta | Yes | Response metadata including pagination, truncation, and usage info |