extract_embedded_state
Use this when a page's data lives in its embedded JavaScript state rather than its rendered HTML - Next.js (NEXT_DATA, and React Server Component payloads with references between rows resolved plus a data_rows index of the rows…
Instructions
Use this when a page's data lives in its embedded JavaScript state rather than its rendered HTML - Next.js (NEXT_DATA, and React Server Component payloads with references between rows resolved plus a data_rows index of the rows carrying data), Nuxt (Nuxt 3's NUXT_DATA decoded), SvelteKit, Apollo, Redux (INITIAL_STATE, PRELOADED_STATE), ytInitialData and ytInitialPlayerResponse, Inertia, Shopify, JSON data-* attributes and blocks. One fetch, exact values, no LLM in the extraction path, so nothing can be fabricated. Payloads are routinely over a megabyte - pass path to return one subtree, keys_only:true to see the first two levels of keys before choosing one, or find:"" to get every path where a field of that name lives, each ready to pass back as path. A result over max_inline_chars comes back as a preview (whole lines of the JSON), data_keys and a result_handle; read the rest with read_result json_path, e.g. "data.next_data.props". Set escalate:true when the site is known to block: the plain fetch still runs first, and only if it is walled does the stealth browser re-read the page, run the same parser and also read the globals off window (window_state: NEXT_DATA, NUXT, ytInitialData and others, read after JavaScript ran) - projected at 7, charged 2 when the plain fetch worked. Not for the rendered text of a page (scrape) or for sites built without a framework payload. Asking again for the same page with different formats, paths or selectors reuses this session's copy for 10 minutes at 1 credit; pass maxAge: 0 to force a fresh fetch. Cost: 2 credits. Example: extract_embedded_state({url: "https://www.ticketmaster.com/discover/concerts", path: "next_data.props.pageProps"})
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| raw | No | Also keep the undecoded __NUXT_DATA__ devalue array under json_scripts, beside the decoded nuxt_data. Default: false | |
| url | Yes | The URL to read embedded state from | |
| find | No | Return `matches` instead of `data`: every property with this key name (case-insensitive) anywhere in the selected data (after `path`), in document order, each as {path, preview} with the first 200 characters of its value - at most 50, with `matches_total` and `matches_truncated`. Each match path already starts with `path`, so it can be passed straight back as `path`. Discover where a field lives without downloading the payload. Cannot be combined with keys_only | |
| path | No | Return only this subtree instead of the whole payload. Dotted keys and array indexes, e.g. "next_data.props.pageProps" or "next_f[0].f" — not JSONPath (no wildcards, filters or recursion). State payloads are routinely over a megabyte; scope them. | |
| maxAge | No | Reuse this session's copy of the page if it was fetched within this many ms (default 600000 = 10 min); 0 always fetches fresh. | |
| escalate | No | When the plain fetch comes back blocked (403/429/challenge page, or an empty shell with no state), re-read the page once in the stealth browser and run the same parser on the rendered document; the browser also reads the framework globals off window (window_state). Under escalate_engine "auto" a Chrome TLS handshake with the honest CrawlForge User-Agent (impit) is tried first, and the browser runs only when that does not get the page. A 404 or 5xx never escalates. Projected at 2+5; the actual charge stays at 2 when the plain fetch succeeded. Default: false | |
| wait_for | No | Escalated render only: extra wait after page load, in ms — for state assigned after DOMContentLoaded. Ignored without escalation | |
| keys_only | No | Return `keys` instead of `data`: the first two levels of keys of the selected data (after `path`), each value replaced by its type ("object", "array(<n>)", "string", "number", "boolean", "null"); an array shows its length and its first item. Cheap discovery before choosing a path. Default: false | |
| user_agent | No | Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with. | |
| respect_robots | No | Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default. | |
| escalate_engine | No | Stealth engine for the escalated retry: "auto" (default — Camoufox when it is installed, Chromium otherwise, reported in warnings), "playwright" (Chromium) or "camoufox" | auto |
| max_inline_chars | No | Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS) |