extract_page_data
Scrape a URL using ScrapingBee and extract specific data using CSS or XPath selectors.
Scope: one URL per call, with the result returned into the conversation.
For many URLs, a whole site, output written to a file or directory,
resumable jobs, or a scheduled re-run, use the ScrapingBee CLI instead —
scrapingbee scrape --input-file urls.txt --output-dir results,
scrapingbee crawl URL --save-pattern ..., scrapingbee schedule --every.
This server has no batch, crawl, file-output or scheduling equivalent.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The URL of the page to scrape. | |
| wait | No | Milliseconds to wait after page load (0–35000). | |
| device | No | desktop (default) or mobile. | desktop |
| cookies | No | Semicolon-separated cookies to send to the target. | |
| timeout | No | Request timeout in milliseconds (1000–140000, default 140000). | |
| max_cost | No | Optional credit ceiling for auto_mode (integer >= 1). The request will not escalate to a configuration costing more than this many credits. 0 means no ceiling. Ignored unless auto_mode is True. | |
| wait_for | No | CSS or XPath selector to wait for before returning. | |
| auto_mode | No | On by default — ScrapingBee automatically picks the cheapest configuration that successfully fetches the page, escalating proxy strength only as needed (you are charged only for the winning config). To choose a configuration yourself instead, use one of: auto_mode=False for the classic tier (no proxy, cheapest), premium_proxy=True, or stealth_proxy=True — the two proxy flags override auto_mode on their own, so auto_mode=False is only needed for the classic tier. Setting render_js also switches to manual. | |
| render_js | No | Whether to use headless-browser rendering. Setting this explicitly switches the request to manual configuration (disables auto_mode). | |
| session_id | No | Integer 0–10000000 to reuse an IP for up to 5 minutes. | |
| ai_selector | No | CSS selector to focus AI extraction on part of the page. | |
| js_scenario | No | Stringified JSON object of browser interaction instructions. | |
| country_code | No | ISO 3166-1 alpha-2 country code (e.g. "fr", "us") to route the request through an IP in that country. Only takes effect when premium_proxy or stealth_proxy is True. | |
| wait_browser | No | domcontentloaded (default), load, networkidle0, networkidle2. | domcontentloaded |
| window_width | No | Viewport width (default 1920). | |
| custom_google | No | Set to True if the url is a google domain in the following format: <subdomain>.google.<top-level-domain> (Example: translate.google.com, mail.google.com, www.google.co.in etc) | |
| extract_rules | Yes | A JSON string defining extraction rules. SIMPLE SYNTAX: Use {"key_name": "css_or_xpath_selector"} format. - CSS selector example: {"title": "h1", "price": "span.price"} - XPath selector example: {"title": "//h1", "items": "//div[@class='item']"} - Extract attribute value of an element using @<attribute> or if you are using extended syntax <element>@<attribute>, for example: {"image": "img@src"}, "link": {"selector": "a","output": "@href"} Note: Selectors starting with "/" are treated as XPath, otherwise CSS. EXTENDED SYNTAX: For more control, use a dict with these options: - "selector": CSS or XPath selector string (required) - "type": "item" (single element, default) or "list" (multiple elements) - "output": It is also possible to add extraction rules inside the output option in order to create powerful extractors. Extended example: { "title" : "h1", "subtitle" : "#subtitle", "articles": { "selector": ".card", "type": "list", "output": { "title": ".post-title", "link": { "selector": ".post-title", "output": "@href" }, "description": ".post-description" } } } | |
| json_response | No | Return the full JSON envelope instead of the raw body. | |
| premium_proxy | No | Manual option: use a premium proxy (middle tier of classic/premium/stealth). Setting this disables auto_mode. | |
| stealth_proxy | No | Manual option: use a stealth proxy (strongest tier of classic/premium/stealth). Setting this disables auto_mode. | |
| window_height | No | Viewport height (default 1080). | |
| block_resources | No | Block images/CSS to speed up rendering. | |
| ai_extract_rules | No | JSON string of AI extraction rules. |
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||