PREFERRED scraping tool. Fetches a page as cheaply as possible: it first tries a FREE direct HTTP request, and only falls back to the PAID Bright Data Web Unlocker if it detects an anti-bot wall (Cloudflare, Akamai, Imperva/Incapsula, PerimeterX, DataDome, a CAPTCHA, or a 403/429/503). An in-memory skip-list remembers hard domains so repeat calls skip the doomed free attempt. Most sites cost $0. The result reports which tier was used and whether credit was spent. Use this instead of unlocker_scrape for read-only fetches; reserve browser_scrape for real JS interaction.

Show the in-memory list of domains currently being sent straight to Bright Data (because the free tier was blocked recently). Useful for debugging cost.

Fetch an array of URLs in parallel using the same free-first routing as smart_scrape. Returns an array of {url, success, text, tier, paid} objects. Use this instead of calling smart_scrape in a loop — it's significantly faster.

Start at a URL, follow links up to a depth/page limit, and return content from each page. Stays on the same hostname by default. Use url_filter to restrict which paths are visited.

Fetch a URL and compare it to the previously cached version. Returns whether the page changed, when it was last seen, and both versions so you can inspect what's different. First call always stores a baseline; subsequent calls detect changes.

Fetch a page and extract structured data: JSON-LD schema.org objects, Open Graph tags, Twitter card meta, standard meta description/keywords, and page title. Returns clean JSON. Useful for getting product info, article metadata, or any schema.org markup without parsing HTML.

Fetch and parse an RSS 2.0 feed, Atom 1.0 feed, or XML sitemap. Auto-detects the format from content-type and content. Returns structured JSON with items/entries or sitemap URLs. Free direct fetch — no Bright Data credit unless the feed is behind a wall.

Fetch and parse the site's robots.txt, then check whether the given URL path is permitted for the wildcard (*) user-agent. Returns the verdict and the matching rule. robots.txt is cached for 24 hours.

Replays an HTTP Archive (.har) file exported from browser DevTools. Automatically detects dynamic values (CSRF tokens, nonces, hidden form fields, JSON token fields) in responses and substitutes fresh values into subsequent requests — so login flows, form submissions, and signed API calls work correctly even though the recorded values are stale. Static assets (images, CSS, JS) are skipped by default. Use 'substitutions' for values that never appear in responses (credentials, API keys). Use 'dry_run' to preview what would be sent.

Always spends Bright Data credit. For read-only fetches prefer smart_scrape — it tries a free direct request first and only falls back here when the site actively blocks it. Use unlocker_scrape directly only when you need screenshot output, want to force Bright Data unconditionally, or smart_scrape has already confirmed the site is hard-blocked.

Start an asynchronous Web Unlocker request for slow/heavy pages. Returns a response_id you later pass to unlocker_get_async_result.

Retrieve the result of an async unlock job. If still processing, returns a PENDING status - poll again shortly.

Return Web Unlocker success-rate statistics (past 7 days) for a domain. Use a wildcard like 'example.*' to get all TLDs.

Run a web search. For DuckDuckGo queries, tries a FREE direct fetch first and only falls back to the paid Bright Data SERP API if blocked. For Google, Bing, and Yandex, always uses the paid SERP API (no free public API exists). Returns structured JSON (parse=true) or raw HTML/Markdown (parse=false).

Trigger a Bright Data dataset (Web Scraper / Crawl API) collection over one or more URLs. Returns a snapshot_id to poll with web_scraper_get_results. Requires a dataset_id (gd_...) - either passed here or via BRIGHTDATA_DATASET_ID.

Check progress and, when ready, download the data for a snapshot_id returned by web_scraper_trigger. If the job is still running, returns its status.

Drive a real, remote cloud browser (Puppeteer over CDP) for JavaScript-heavy sites and multi-step flows. Supports automatic CAPTCHA solving, ad blocking, device emulation, sticky sessions, clicks/typing/scrolling, and returns rendered HTML, inner text, and/or a full-page screenshot. No local browser is required - everything runs on Bright Data's infrastructure. Requires BRIGHTDATA_BROWSER_AUTH to be set.

Structured Amazon product data (title, price, ratings, images, specs). Requires a product URL containing /dp/.

Structured Amazon review data for a product. Requires a product URL containing /dp/.

Structured Amazon search results for a keyword. Returns first page.

Structured Walmart product data. Requires a product URL containing /ip/.

Structured Walmart seller profile data.

Structured eBay listing data (price, condition, seller, bids).

Structured LinkedIn job listing data.

Structured Instagram profile data (bio, followers, following, post count).

Structured Instagram post data (caption, likes, comments, media).

Structured Instagram reel data.

Structured Instagram comments for a post.

Structured Facebook post data.

Structured Facebook Marketplace listing data.

Structured TikTok profile data (bio, followers, following, video count).

Structured TikTok video data (views, likes, shares, comments).

Structured TikTok comment data for a video.

Structured YouTube channel data (subscribers, description, videos).

Structured YouTube video data (views, likes, description, tags).

Structured YouTube comment data for a video.

Structured X/Twitter post data (text, likes, retweets, replies).

Structured Reddit post data (title, body, upvotes, awards, subreddit).

Structured Reddit comment thread for a post.

Structured LinkedIn person profile data (headline, experience, education, skills).

Structured LinkedIn company profile data (industry, size, description, specialties).

Structured LinkedIn post/article data.

Structured Glassdoor company profile (rating, CEO approval, culture scores).

Structured Glassdoor employee reviews for a company.

Structured ZoomInfo company data (employees, revenue, technologies, contacts).

Structured Crunchbase company data (funding rounds, investors, founded, HQ).

Structured Zillow property data (price, beds, baths, sqft, listing details).

Structured GitHub repository or file data (stars, forks, contributors, file content).

Structured Reuters news article data (headline, body, author, published date).

Structured Google Play app data (rating, downloads, reviews, description).

Structured Apple App Store app data (rating, reviews, price, description).

Structured Booking.com hotel data (price, rating, amenities, location).

Query ChatGPT and get structured AI-generated insights about a brand or topic.

Query Grok and get structured AI-generated insights about a brand or topic.

Query Perplexity and get structured AI-generated insights about a brand or topic.

Structured npm package data (latest version, README, dependencies, weekly downloads).

Structured PyPI package data (latest version, README, dependencies, metadata).