crawl_site
Crawl a site and get clean markdown for every page in one synchronous call (no polling). Same-host BFS with depth and page-budget limits, robots.txt honoured by default. Price: $0.0015 per requested page (1-25) — you are billed for the requested budget, not the pages returned
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Start URL — the crawl stays on this host | |
| limit | No | Page budget (1-25, default: 10) — charged per requested page | |
| exclude | No | Skip URLs whose path+query contains one of these substrings | |
| include | No | Only crawl URLs whose path+query contains one of these substrings | |
| timeout | No | Per-page timeout in ms (default 10000) | |
| maxChars | No | Per-page content character cap (default 8000) | |
| maxDepth | No | Link depth from the start URL (0-3, default: 2) | |
| respectRobots | No | Honour robots.txt (default: true; disable only for sites you control) |