kloakt_crawl
Breadth-first crawl of a domain starting from a given URL, collecting clean page content up to a configurable page and depth limit. Stays on the same domain by default.
Instructions
Breadth-first crawl from a start URL, returning clean content for each page visited. Budget-limited (max_pages) and depth-limited (max_depth), stays on the start domain by default, with cycle detection. Use to gather a small section of a site in one call. Network-bound — keep max_pages modest.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| stealth | No | Enable anti-detection mode | |
| max_chars | No | Truncate each page's content to N chars (default: 2000) | |
| max_depth | No | Max link-hops from the start URL (default: 2) | |
| max_pages | No | Hard cap on pages to fetch (default: 10) | |
| start_url | Yes | Where to begin crawling | |
| same_domain | No | Restrict crawl to the start URL's host (default: true) |