scraper_crawl_url
Crawl multiple web pages from a starting URL using BFS link discovery and return compressed markdown with 70-90% fewer tokens than raw HTML.
Instructions
Crawl multiple pages from a starting URL using BFS link discovery. Returns compressed markdown for each page with 70-90% fewer tokens than raw HTML.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Starting URL to crawl | |
| maxPages | No | Max pages to crawl (default: 10) | |
| maxDepth | No | Max link depth (default: 2) | |
| mode | No | Fetch mode: 'fast' (plain HTTP), 'stealth' (TLS fingerprint), 'render' (headless browser), 'auto' (fast with fallback). Default: 'auto' | |
| include | No | URL patterns to include (glob) | |
| exclude | No | URL patterns to exclude (glob) | |
| timeout | No | Per-page timeout in milliseconds (default: 10000) |