start_crawl
Start a bounded breadth-first crawl from a starting URL and receive a job ID immediately, which you can use to poll crawl status and retrieve results asynchronously.
Instructions
Start a bounded breadth-first crawl. Returns a job_id immediately (non-blocking).
The crawl follows in-scope links from start_url up to max_pages / max_depth.
allow_patterns/deny_patterns: regexes a link URL must match / must not match.same_domain(default true): restrict the crawl tostart_url's host.selectors: same format as theextracttool; when given, each crawled page yields the extracteddata. Otherwise each page yields a short text excerpt.obey_robots,download_delay: per-crawl overrides of the server defaults.
Poll progress with crawl_status(job_id) and read items with crawl_results(job_id).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| max_depth | No | ||
| max_pages | No | ||
| selectors | No | ||
| start_url | Yes | ||
| obey_robots | No | ||
| same_domain | No | ||
| deny_patterns | No | ||
| allow_patterns | No | ||
| download_delay | No |