web_crawl
Crawl a bounded set of internal pages from a single public website, respecting robots rules and hard limits on depth and page count.
Instructions
Crawl a bounded set of internal pages from one public website.
Uses Crawl4AI breadth-first crawling. External links are not followed. The server enforces hard caps on depth/pages regardless of the requested values.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| max_depth | No | ||
| max_pages | No | ||
| per_page_chars | No | ||
| respect_robots | No |