scrape
Fetch a single web page through a residential proxy and return clean Markdown, escalating to a real browser when bot-protected. Supports structured or AI extraction for targeted data.
Instructions
Scrape a single web page through a residential proxy and return it as clean Markdown (or HTML/text). Uses a real Chrome TLS fingerprint by default and only spins up a headless browser if the page is bot-challenged. Optionally run structured extraction (CSS selectors) or AI extraction (natural-language prompt). Markdown keeps the complete page by default (content_mode 'smart': everything except nav/footer/cookie chrome, with GFM tables and absolutized links); to inspect a page's raw no-JS/SEO fallback use format 'html'.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| toc | No | Prepend a table of contents built from the page headings | |
| url | No | The page URL to scrape (optional only when you pass `html` to convert) | |
| xhr | No | Record the page's XHR/fetch traffic (URL, method, status, response body) into payload.xhr. Forces a browser render. An SPA's own JSON API is usually far cleaner than its DOM — use this to DISCOVER the API, then fetch_resource to return it directly. | |
| html | No | Convert HTML you already have instead of fetching: no proxy bandwidth is used, and the full parser pipeline still applies. Pass `url` too if you want relative links absolutized. | |
| mode | No | summary: return only metadata (title, description, canonical, contentLength, status, engine, bytes) with no page content — use this when auditing pages instead of reading them | |
| chunk | No | Segment the output into payload.chunks[] for RAG/vector-DB ingestion — each chunk carries its heading path and token count. Fences and tables are never split. | |
| query | No | What you are looking for on the page. Keeps only the relevant sections (BM25 scoring over blocks, headings preserved) — the way to read one fact off a huge page without spending its whole token budget. | |
| engine | No | auto (default): TLS tier, escalate to browser on block. tls: never escalate — exactly what a pure HTTP bot (no JS) sees, right for SEO checks. render: force browser. | |
| format | No | Output format (default markdown) | |
| parser | No | Your own parsing rules, as CSS selector lists — use these when you know the page and don't want to rely on heuristics. include: keep ONLY these subtrees (targeted extraction, e.g. ['article.post']). exclude: delete site-specific chrome we kept. keep: protect a section (sidebar, dialog, form) that smart mode would strip. | |
| render | No | Force the headless browser (JS execution) | |
| actions | No | Ordered browser interactions before capture (forces a render). Each is one object: {"click":"#sel"}, {"clickText":"Accept"} (click by visible text — dismiss a consent wall without knowing its CSS), {"type":{"selector":"#q","text":"shoes"}}, {"scroll":"bottom"}, {"wait":1000}, {"waitForSelector":".results"}. Add "optional":true to skip a miss, or "timeoutMs":N to bound one action. | |
| cookies | No | Cookies to send as name→value — the simple way to scrape behind a login | |
| country | No | ISO country code for the proxy exit, e.g. 'us' | |
| extract | No | Structured-extraction schema: { field: "css selector" | { selector, attr, all, fns } }. `fns` is a transform pipeline run on the value — e.g. { "price": { "selector": ".price", "fns": ["amount_from_string"] } } returns a number, not text. Functions: amount_from_string, amount_range_from_string, convert_to_float/int/str, trim, lower, upper, {regex_search|regex_find_all: "pat"}, {replace:{from,to}}, {join:","}, {select_nth:0}, length, unique, max, min, average, product. | |
| formats | No | Additional formats to return together in payload.formats, e.g. ['markdown','text'] | |
| ai_prompt | No | Natural-language instruction — the LLM turns the page into structured JSON | |
| ai_schema | No | JSON Schema for deterministic AI extraction; returned under payload.ai.data | |
| app_state | No | Mine the page's own hydration state (Next.js __NEXT_DATA__, Nuxt, embedded JSON islands) into payload.metadata.appState. This is where SPAs keep the real data — prices behind a picker, stock, download counts, listings — even when the DOM shows only a shell, so it often answers the question without a browser render. true/'auto': pruned to the informative parts (recommended). 'raw': the complete blobs, up to 512KB. | |
| preset_id | No | Run a stored parser preset (see save_parser_preset) instead of passing `extract` selectors. Results land in payload.data exactly the same way, and the run is scored so the preset can detect decay and self-heal. | |
| highlights | No | With `query`: also return the N most relevant passages in payload.highlights | |
| links_mode | No | Link rendering. inline (default): [text](url). footnote: URLs moved to a numbered reference list at the end. strip: keep only the link text — cuts 30-48% of the tokens on link-dense pages when you only need the prose. | |
| max_tokens | No | Cap the markdown at ~this many tokens, cutting at a section boundary (never inside a table or code block) and noting how much was omitted | |
| frontmatter | No | Prepend YAML front-matter (title, url, canonical, description, author, date) so the markdown is self-contained for RAG/Obsidian pipelines | |
| images_mode | No | inline (default) keeps ; 'alt' keeps only alt text; 'strip' removes images | |
| content_mode | No | smart (default): whole page minus nav/footer/cookie chrome. article: Readability main article only (news/blogs). full: entire body as-is. | |
| content_modes | No | Return several content scopes from ONE fetch under payload.contents (e.g. compare smart vs full) | |
| include_links | No | Return all de-duplicated absolute page links in payload.links | |
| reveal_hidden | No | Render tier only: before capturing, open <details>/accordions and click through every tab, appending each revealed panel to the page. Use it for tabbed code samples or spec accordions where a plain render captures only the visible variant. | |
| fetch_resource | No | Regex matched against the page's network requests: the first matching response's BODY becomes the result instead of the page HTML (e.g. '/api/products' to get an SPA's JSON directly). Forces a render. Fails with 504 if nothing matches. | |
| summary_sections | No | Append 'Links on this page' / 'Images on this page' sections — handy when deciding the next hop |