crawl
Start an asynchronous breadth-first site crawl from a seed URL and convert pages to Markdown. Get a job ID to poll for completion status.
Instructions
Start an asynchronous BFS crawl of a site from a seed URL, converting each page to Markdown. Returns a job id — poll with crawl_status.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Seed URL | |
| depth | No | Max link depth (default 3) | |
| limit | No | Max pages (default 50) | |
| country | No | ISO country code for the proxy exit | |
| exclude | No | URL substrings/globs to exclude | |
| include | No | URL substrings/globs to include | |
| content_mode | No | Per-page content scope: smart (default) | article | full |