Scrape a URL
scrapeFetch a single URL and return selected page content—markdown, HTML, links, images, screenshots, or metadata—for one-shot extraction; use CSS selectors to pull named values like prices and titles.
Instructions
Fetch a single URL via the crawlbrulee scraping API and return the requested content (markdown, cleaned HTML, raw HTML, links, images, screenshot, page metadata in metadata). Use this for one-shot page extraction. For full-site discovery use the map tool first. To pull named values such as prices, titles or links, set extract.elements (name → CSS selector) instead of reading the whole markdown: they come back in elements, for no extra credits. Rules: https://crawlbrulee.com/docs/scrape/elements. Check page_status_code before you trust the content: it is the HTTP status the site answered with for the final page. A page the site served is a successful result whatever its status, so a 404 or 503 page comes back with its content, not as an error. A 404 means the markdown is the site's "not found" page, not the page you asked for. 2xx and 4xx pages are billed, except 403, 407, 408, 429 and 451; 5xx pages are never billed. If the site cannot be reached at all the tool returns a target_unreachable error (HTTP 502, not billed): retry later or check the URL. Screenshot URLs in the response are signed download links that expire 24 hours after the scrape: download the image if you need it later; do not keep the link. The response also carries response_meta.usage: total_credit_cost = engine_credit_cost × proxy_multiplier + screenshot_slicing_credit_cost + zero_data_retention_credit_cost, plus engine and proxy (the resolved tier — never auto). engine is the billed engine (http, browser, screenshot, cache); a cache hit is represented by engine: "cache". Set zero_data_retention: true to keep the result out of the shared cache; anything stored to deliver it is kept for 24 hours, then deleted. It adds 1 credit and must be enabled for the organization (else a zero_data_retention_not_enabled error, not billed). See https://crawlbrulee.com/docs/zero-data-retention. Extraction is capped per page: 30,000 links, 10,000 inline images and 10,000,000 characters of HTML. A page past a cap is truncated rather than refused, and the response warnings array names which one (links_truncated, inline_images_truncated, raw_html_truncated) — so treat that output as incomplete. elements_truncated means an elements value hit a limit. Use cleanup to control what is removed before any output is built: ads_and_popups (on by default) drops ads, cookie banners and chat widgets, and exclude_selectors removes anything else. It shapes markdown, cleaned_html, links, images and the screenshot, and NEVER raw_html — so request raw_html when you need the page exactly as it arrived. warnings also reports a section whose extraction failed outright (links_unavailable, inline_images_unavailable, metadata_unavailable): that field comes back omitted or empty while the rest of the scrape succeeded, so do NOT conclude the page had no links/images/metadata — re-run the scrape instead. An empty field with no such warning does mean the page had none. screenshot_unavailable means a screenshot was asked for, but the page came back from the http engine without one.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL to scrape. Known tracking parameters are removed before the page is fetched, so they are neither sent to the target site nor part of the cache key. Every other query parameter is kept verbatim and is part of the cache key. | |
| cache | No | Cache settings for this request | |
| proxy | No | Proxy tier to use for fetching | auto |
| cleanup | No | What is removed from the page before any output is built. Applies to markdown, cleaned_html, links and images on every engine, and to the screenshot. Never applies to raw_html, which is always the page before we removed anything. | |
| extract | No | Which content formats to extract. Defaults to metadata + cleaned_html. | |
| location | No | Optional locale + country emulation for the scrape | |
| require_js | No | Use a headless browser to render JavaScript before scraping | |
| zero_data_retention | No | Keeps the result out of the shared cache; anything stored to deliver it is kept for 24 hours, then deleted. Adds 1 credit. Must be enabled for your organization. See https://crawlbrulee.com/docs/zero-data-retention. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL that was actually scraped, after any redirects, in normalized form — the base that links, images, and internal labels are computed against | |
| links | No | Links found on the page | |
| images | No | Inline images found on the page | |
| elements | No | The values asked for in `extract.elements`, under the same names. Each name is a string, an object (a spec with `fields`), a list of either (`all: true`), or null when nothing matched (`[]` with `all: true`). An object holds its field names with the same kinds of values, nested up to 3 levels. Every requested name is always present, at every level of `fields`. | |
| markdown | No | Page content converted to clean Markdown | |
| metadata | No | Extracted page metadata (title, OG tags, etc.) | |
| raw_html | No | Raw, unprocessed HTML of the page | |
| warnings | No | Non-error notices about the scrape. Truncation codes — `screenshot_truncated` (long page exceeded the scrolling-screenshot height cap), `links_truncated` / `inline_images_truncated` (page had more links/images than the per-page extraction caps), `raw_html_truncated` (rendered HTML exceeded the per-page size budget) — mean the field is present but capped. `elements_truncated` means at least one `elements` value hit a limit: a list was cut at 1,000 matches, the request reached 10,000 matches in total, or a value did not fit the ~1 MB of values allowed per request (that value is then null, never cut short). Unavailability codes — `links_unavailable` / `inline_images_unavailable` / `metadata_unavailable` — mean that optional field could not be extracted and was omitted (null/empty) while the rest of the scrape succeeded, so an empty field carrying one of these does NOT mean the page had none. `screenshot_unavailable` means a screenshot was asked for, but the page came back from the http engine without one. Stable string codes — clients can switch on them. Warnings are stored with the result: async result fetches and cache hits carry them too, filtered to the fields the request asked for. `metadata_truncated` is retired and no longer sent: metadata has no size limit of its own now, though the code can still appear on results stored before that change. | |
| screenshot | No | Screenshot of the page, if requested | |
| cleaned_html | No | Cleaned HTML of the main page content | |
| content_type | No | Content-Type header returned by the server | |
| requested_url | Yes | The URL you requested, echoed verbatim — before any redirects | |
| response_meta | Yes | Request-level metadata. `response_meta.usage` reports what the request cost (total_credit_cost and its parts), the billed engine and the resolved proxy tier. | |
| page_status_code | No | The HTTP status the site answered with for the final page, after redirects. Check it before you trust the content. A page the site served is a successful result whatever its status: a 404 or 503 page comes back with its content here, not as an error, so a 404 means the markdown is the site's "not found" page, not the page you asked for. Billing follows this status: 2xx and 4xx pages are billed, except 403, 407, 408, 429 and 451; 5xx pages are never billed. Older api versions do not send it; they only returned pages that loaded normally. | |
| unsupported_fields | No | Extract fields that were requested but are not supported for this content type |