crawl_site
Extract content from websites by crawling with configurable depth, page limits, request budgets, and per-host rate limiting.
Instructions
Crawl a site with strict request/error/token budgets and per-host rate limiting.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| compact | No | ||
| delay_ms | No | ||
| max_depth | No | ||
| max_pages | No | ||
| start_url | Yes | ||
| max_errors | No | ||
| concurrency | No | ||
| render_mode | No | auto | |
| max_requests | No | ||
| output_format | No | markdown | |
| exclude_pattern | No | ||
| include_pattern | No | ||
| max_total_chars | No | ||
| include_documents | No | ||
| include_subdomains | No | ||
| max_chars_per_page | No | ||
| deduplicate_content | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||