alterlab_scrape
Scrape any website into LLM-ready markdown or structured JSON while bypassing anti-bot protection, rendering JavaScript, and handling dynamic pages.
Instructions
Get data from any website, bypass Cloudflare and anti-bot protection, scrape JavaScript-rendered pages, or fetch content from dynamic single-page apps. Turn any URL into clean, LLM-ready markdown — or get text, HTML, JSON, and structured sections. Automatically bypasses anti-bot protection (Cloudflare, Akamai, DataDome, PerimeterX, hCaptcha) with intelligent tier escalation — no manual configuration needed. Cost-efficient: tier 1 (curl) is $0.0002/page, tier 2 (http) $0.0003, tier 3 (stealth) $0.002, tier 3.5 (lightjs) $0.0025, tier 4 (browser) $0.004; you only pay for the tier that succeeds. Add-ons: screenshot +$0.0002, generate_pdf +$0.0004, ocr +$0.001, use_proxy +$0.0002, LLM extraction +$0.001, embeddings +$0.001; method='POST' costs 1.5x. Returns markdown by default — optimized for LLM context. Supports GET (default) and POST via the method parameter. Use method='POST' with body for GraphQL APIs, REST endpoints, and form submissions. Use content_type to set the POST body Content-Type (json, urlencoded, graphql, plain). Use headers to send custom HTTP headers to the target. Use render_js=true to scrape dynamic pages, JavaScript-heavy sites (React, Angular, Vue, SPAs). Use render_js='auto' for mixed sites to detect JS needs per-page (saves 30-60%). Use actions (click, fill, press, scroll, wait_for, evaluate, ...) to drive the page before extraction. Use use_proxy=true for geo-restricted or heavily protected sites; use_own_proxy for your own proxy integration. Use sticky_session to reuse cookies and exit IP across calls. Use skip_tls_verification only for sites with broken certificates. Use formats=['json_v2'] for a structured section tree (headings + content blocks). Use formats=['rag'] for chunked text optimized for RAG pipelines. Use formats=['raw'] for the raw response body without extraction. Use formats=['content'] for AI/KB pipelines — returns body_markdown, content_hash, images, links. Use extraction_schema to extract structured fields from the page using LLM. Use extraction_prompt for natural language extraction instructions. Use extraction_profile or extraction_template for pre-built templates (product, article, job_posting, etc.). Use evidence=true to include source passages alongside extracted fields. Use embeddings to chunk and embed the scraped text. Use cache=true and cache_ttl to enable response caching. Use cost_controls to cap spending, pin a tier, or set a time budget. Supports authenticated scraping via session_id, inline cookies or session_headers. Use scroll_to_load=true for infinite-scroll pages. Use location.country to scrape geo-targeted content from any region. Use prefer_cost=true to minimize credit spend (starts from cheapest tier). Use prefer_speed=true to skip to a fast reliable tier immediately. Use fail_fast=true to error instead of auto-escalating to expensive tiers. Use force_refresh=true to bypass cache and always fetch live content. Use promote_schema_org=true to prefer Schema.org JSON-LD over LLM extraction on structured pages. Use estimate_first=true to run a free cost estimate before scraping (prepended to the result). If a long-running scrape returns a job_id instead of content, the job continues server-side.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| ocr | No | Extract text from images using OCR (+$0.001) | |
| url | Yes | URL to scrape | |
| body | No | Request body for POST requests. For GraphQL: JSON string with 'query' and optional 'variables' fields (e.g., '{"query": "{ user { id name } }"}').For REST APIs: JSON-encoded payload string. For form submissions: URL-encoded key=value pairs (e.g., 'name=Alice&email=alice@example.com'). Omit for GET requests. | |
| mode | No | Scraping mode: auto (recommended), html, js (headless browser), pdf, or ocr | auto |
| cache | No | Enable caching for this request. When true, repeat requests with identical parameters may return cached results. Only use for idempotent requests (GET pages, read-only POSTs). | |
| method | No | HTTP method for the request. Default GET (standard page scraping). Use POST for GraphQL endpoints, form submissions, and REST API calls. When using POST, provide body with the request payload. POST costs 1.5x base tier price. | GET |
| actions | No | Browser automation actions run in order after page load (max 20, requires render_js=true). Failed actions are skipped without aborting the sequence. Results are returned in action_results. Each action has a 'type': wait, wait_for, click, fill, press, scroll, infinite_scroll, screenshot, evaluate. | |
| cookies | No | Inline cookies as key-value pairs for authenticated scraping (e.g., {"session_token": "abc123"}). Use this for one-off requests; use session_id for reusable sessions. | |
| formats | No | Output formats. 'markdown' is best for LLM consumption. 'json_v2' returns a structured section tree (headings + content blocks). 'rag' returns chunked text optimized for retrieval-augmented generation. 'raw' returns the raw response body without extraction. 'content' returns body_markdown + content_hash + images + links for AI/KB pipelines. | |
| headers | No | Custom HTTP headers to send to the target URL. Hop-by-hop headers (Connection, Transfer-Encoding, Host, etc.) are not allowed. | |
| timeout | No | Request timeout in seconds (1-300) | |
| evidence | No | Include provenance/evidence snippets alongside extracted fields. Each extracted value will include the source text passage it was derived from. Requires extraction_schema or extraction_prompt. | |
| location | No | Geo-targeting parameters for localized content scraping. Controls proxy country routing, Accept-Language header, and browser locale. | |
| template | No | DEPRECATED alias for extraction_template, kept for backward compatibility. Sent to the API as extraction_template; prefer extraction_template. | |
| wait_for | No | CSS selector to wait for before extracting content (e.g., '#main-content') | |
| cache_ttl | No | Cache TTL in seconds (60–86400). Defaults to 3600 (60 min) when cache=true. Requires cache=true. | |
| fail_fast | No | Fail immediately if the page requires an expensive tier (browser/captcha) instead of auto-escalating. Shorthand for cost_controls.fail_fast. Returns an error with the required tier instead of upgrading. | |
| render_js | No | Render JavaScript using a headless browser (forces Tier 4 minimum, $0.004/req — no separate add-on charge). Required for JS-heavy sites. Set to 'auto' for smart detection (probes each page, only renders JS-heavy pages with a browser — saves 30-60% on mixed sites). | |
| use_proxy | No | Route through AlterLab's premium proxy (+$0.0002). Helps bypass geo-restrictions and anti-bot | |
| embeddings | No | Chunk the scraped text and embed it with OpenAI models (+$0.001 per page). Set enabled=true; model, chunk_size (64-8191) and chunk_overlap (0-2048) are optional. | |
| pdf_format | No | Output format for mode='pdf' extraction (API default 'markdown') | |
| screenshot | No | Capture a full-page screenshot (+$0.0002, requires render_js). The URL is returned as screenshot_url. Use alterlab_screenshot to get the image inline. | |
| session_id | No | UUID of a stored session for authenticated scraping. Use alterlab_list_sessions to find available sessions. The session's cookies will be injected into the request. | |
| wait_until | No | Page load event to wait for before extracting (networkidle, domcontentloaded or load; API default networkidle) | |
| prefer_cost | No | Optimize for lowest cost — try cheaper tiers first before escalating. Shorthand for cost_controls.prefer_cost. Mutually exclusive intent with prefer_speed. | |
| sync_policy | No | Synchronous wait policy. 'auto' (default) may return an early 202 with a job_id for predicted T3/T4 jobs; 'wait_until_deadline' keeps the connection open until the terminal result or the safe sync deadline (~85s). | |
| block_images | No | Block image downloads during browser rendering. Reduces proxy bandwidth and speeds up scrapes. Only effective with render_js=true. | |
| content_type | No | Content-Type header for the request body. Defaults to 'application/json' when body is provided. Use 'application/graphql' for raw GraphQL queries. Use 'application/x-www-form-urlencoded' for HTML form submissions. Requires body to be set. | |
| generate_pdf | No | Generate a PDF of the rendered page (+$0.0004, requires render_js) | |
| ocr_language | No | Tesseract language code for OCR (API default 'eng') | |
| prefer_speed | No | Optimize for speed — skip to a reliable tier immediately instead of escalating from Tier 1. Shorthand for cost_controls.prefer_speed. Mutually exclusive intent with prefer_cost. | |
| scroll_count | No | Number of scroll iterations when scroll_to_load is enabled (1-10, API default 3) | |
| cost_controls | No | Fine-grained cost and tier controls. Use to cap spending, pin a tier, or trade off cost vs speed. | |
| enable_scroll | No | Force scrolling for lazy-load images. true: always scroll (+5-10s), false: never scroll, omit: auto (scroll unless social media site). | |
| force_refresh | No | Bypass the cache and always fetch a fresh copy of the page. Use when you need real-time content and a cached result would be stale. | |
| proxy_country | No | ISO country code for proxy geo-targeting (e.g., 'US', 'DE'). Requires use_proxy=true | |
| use_own_proxy | No | Use your own proxy integration (bring-your-own-proxy) instead of the system proxy. BYOP requests get a 20% discount on the tier price. | |
| estimate_first | No | Run a cost estimate before scraping and include it in the response. Adds one lightweight API call (~50ms) with no credit charge. The estimated tier, cost, and confidence are prepended to the scrape result. Useful for unfamiliar or potentially expensive sites — see cost before committing. | |
| filter_content | No | Apply quality filtering to extracted content. When false (default), returns all parsed content without quality thresholds (lossless mode). When true, filters low-quality boilerplate. | |
| scroll_to_load | No | Scroll page to trigger lazy-loaded content (requires render_js). Performs explicit viewport-height scrolls to load dynamic content. Adds ~2-3s latency. | |
| section_filter | No | Filter for the json_v2 section tree. Only applies when 'json_v2' is in formats. | |
| sticky_session | No | Sticky-session handle. Scrapes that pass the same value reuse the cookie jar captured on the first call and, best-effort, the same exit IP (~10 min). Alphanumeric, '-' and '_' only. Requires a paid plan (growth tier or above). | |
| wait_condition | No | Wait condition for JS rendering: domcontentloaded, networkidle or load | |
| include_iframes | No | Inline iframe content into the main document (free, requires render_js). Same-origin iframes are read directly; cross-origin iframes are marked but not fetched. | |
| session_headers | No | Inline auth headers sent to the target (e.g. Authorization: Bearer ...). Merged with the stored session's headers when session_id is also set. | |
| extraction_model | No | Per-request LLM model override in provider-specific format (e.g. 'gpt-4o', 'claude-opus-4-5-20251101', 'llama3-70b-8192'). Overrides the model saved in your BYOK key settings for this request only. | |
| include_raw_html | No | Include raw HTML in the response alongside formatted content | |
| use_system_proxy | No | Override your default proxy integration and use AlterLab's system proxy for this request. | |
| extraction_prompt | No | Natural language extraction instruction. Describes what fields to extract from the page. Mutually exclusive with extraction_schema. Example: "Extract the product name, price, and availability". | |
| extraction_schema | No | JSON schema for structured extraction. The API extracts fields matching this schema from the scraped page using LLM (+$0.001). Result is returned in extraction_result. Example: { "title": "string", "price": "number", "in_stock": "boolean" } | |
| extraction_profile | No | Pre-built extraction schema template. auto: detect best template (API default). product: e-commerce product details. article: news/blog article fields. job_posting: job listing fields. faq: FAQ entries. recipe: recipe ingredients and instructions. event: event details. ecommerce_homepage: homepage product listings. directory_listing: directory/listing page entries. Mutually exclusive with extraction_template. | |
| flatten_shadow_dom | No | Flatten Shadow DOM roots into regular DOM for extraction (free, requires render_js). | |
| max_response_bytes | No | Soft cap on raw response body size in bytes. When the downloaded HTML exceeds this value it is truncated before extraction. Default: 5 MB (5242880). Set to 0 for no limit. Maximum: 50 MB (52428800). Useful for very large pages where you only need the beginning of the content. | |
| promote_schema_org | No | Use Schema.org JSON-LD/Microdata as the primary structured-data source when present (API default true). Promotes machine-readable metadata embedded in the page over LLM extraction. Most effective on e-commerce, recipe, and news article pages. | |
| sticky_session_ttl | No | Sticky-session cookie-jar lifetime in seconds (300-7200, API default 1800). Ignored unless sticky_session is set. | |
| extraction_provider | No | LLM provider to use for extraction. Selects the matching BYOK key registered at /dashboard/settings/llm-keys. When omitted, the most recently used registered key is used automatically. Requires extraction_schema or extraction_prompt. | |
| extraction_template | No | Shorthand alias for extraction_profile — selects the same pre-built template. Mutually exclusive with extraction_profile. | |
| proxy_integration_id | No | Specific proxy integration ID to use (requires use_own_proxy=true) | |
| remove_cookie_banners | No | Remove cookie consent banners from HTML before content extraction (free; the API default is true) | |
| skip_tls_verification | No | TLS certificate verification for the target. Omit (recommended): certificates are verified, and if the site's own certificate is invalid (expired, self-signed, wrong hostname, incomplete chain) the page is still fetched without verification. true skips verification from the start. false always requires a valid certificate, so sites with an invalid certificate fail. |