Start a site crawl
crawlStart an async crawl of one website: pages read as Markdown, following links up to max_depth hops (and the site's sitemaps), honouring robots.txt. With query, the most relevant pages are read first. Returns a task id; call crawl_result with it. 1 credit per page read; limit credits are reserved and the rest refunded.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Start page | |
| limit | No | Maximum pages (default 25) | |
| query | No | Read the pages most relevant to this first | |
| max_depth | No | Link hops from url (default 2) | |
| max_tokens | No | Cap per page | |
| exclude_paths | No | Regular expressions; never follow URL paths matching one | |
| include_paths | No | Regular expressions; follow only URL paths matching one |