Web Scraping API
web_scrapeScrape any URL with rotating proxies and optional JavaScript rendering, returning HTML or AI-extracted data.
Instructions
General-purpose web scraper. Fetches any URL through Scrapingdog's rotating proxies with optional JavaScript rendering, premium proxies, geo-targeting, and AI-based extraction. Returns the page HTML (or extracted data).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The absolute URL of the page to scrape. | |
| wait | No | Milliseconds to wait after page load before capturing (only with dynamic=true). | |
| country | No | Two-letter ISO country code for the proxy geo-location (requires premium). (API default: us) | |
| dynamic | No | Set true to render JavaScript with a headless browser. Costs more credits. (API default: false) | |
| premium | No | Set true to use premium (residential) proxies for hard-to-scrape sites. (API default: false) | |
| ai_query | No | Natural-language instruction to extract specific data from the page via AI. | |
| markdown | No | Return the page content converted to clean markdown instead of raw HTML. (API default: false) | |
| session_number | No | Reuse the same proxy IP across requests by passing a stable integer. | |
| ai_extract_rules | No | JSON string of field->instruction rules for structured AI extraction. |