batch_extract_web_content
Extract token-optimized Markdown from up to 20 web pages in parallel for docs, product lists, or source comparisons. Returns per-URL results and aggregate token counts.
Instructions
Batch extract token-optimized Markdown from multiple website URLs concurrently, with aggregate token statistics.
What it does Runs the same fast/deep pipeline as extract_web_content across an array of up to 20 URLs in parallel, then returns per-URL results plus a summary (urls attempted, succeeded, failed, total tokens, and per-page token counts).
How to use
Use when you need many pages at once — docs, a list of product pages, a set of sources to compare.
mode applies to every URL. concurrency (1-10) caps parallel workers; keep it modest on fast machines / low bandwidth.
css_selector is applied to all URLs. headers/cookies/proxy apply globally to every request.
Each URL is independent: a single failure does not abort the batch (results carry an "error" field for that URL).
Returns Structured JSON with per-url markdown/tokens/error and aggregate totals. Incomplete or empty batches still return a valid result object.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| mode | No | Crawl mode: 'fast' (native HTTP fetch) or 'deep' (headless Playwright browser). | fast |
| urls | Yes | Array of target website URLs to extract (1-20). | |
| proxy | No | Optional HTTP/SOCKS5 proxy URL. | |
| cookies | No | Optional custom HTTP cookies key-value dictionary. | |
| headers | No | Optional custom HTTP request headers key-value dictionary. | |
| concurrency | No | Maximum parallel HTTP/browser crawl worker concurrency (default: 3). | |
| max_retries | No | Maximum retry attempts per URL (default: 3). | |
| css_selector | No | Optional CSS selector to filter DOM node across all target URLs. |