Scrape a URL (cost-aware: free first, Bright Data only if blocked)
smart_scrapeFetches web pages via free direct requests first, automatically escalating to Bright Data's paid unlocker only when anti-bot walls block access, minimizing scraping costs.
Instructions
PREFERRED scraping tool. Fetches a page as cheaply as possible: it first tries a FREE direct HTTP request, and only falls back to the PAID Bright Data Web Unlocker if it detects an anti-bot wall (Cloudflare, Akamai, Imperva/Incapsula, PerimeterX, DataDome, a CAPTCHA, or a 403/429/503). An in-memory skip-list remembers hard domains so repeat calls skip the doomed free attempt. Most sites cost $0. The result reports which tier was used and whether credit was spent. Use this instead of unlocker_scrape for read-only fetches; reserve browser_scrape for real JS interaction.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Full target URL including https:// | |
| method | No | HTTP method. | GET |
| country | No | 2-letter exit-IP country (only applies if it escalates to Bright Data). | |
| headers | No | Custom request headers for both tiers. | |
| free_only | No | Never spend credit; return best-effort free result even if blocked. | |
| use_cache | No | Return a cached response if one exists and isn't expired. | |
| data_format | No | Output format. 'content_only' strips nav/ads and returns just the main article text. | markdown |
| cache_ttl_ms | No | Cache TTL in ms (default 1 hour). Only used when use_cache=true. | |
| rate_limit_ms | No | Minimum ms between free-tier requests to the same domain (0 = off). | |
| wait_for_text | No | Text to wait for (Bright Data tier only). | |
| min_body_bytes | No | Treat a 2xx response smaller than this as a block (0 = off). | |
| respect_robots | No | Check robots.txt before fetching. Returns an error if the path is disallowed. | |
| ignore_skip_list | No | Ignore the hard-domain skip-list and always try the free tier first. | |
| direct_timeout_ms | No | Abort the free fetch after this many ms, then escalate. | |
| force_bright_data | No | Skip the free attempt and go straight to the paid Unlocker. | |
| wait_for_selector | No | CSS selector to wait for (Bright Data tier only). |