crawl_create
Crawl a start URL and follow its links up to a depth, keeping only URLs that match your include and exclude patterns. Returns crawl_id for asynchronous results; set max_pages to bound billing.
Instructions
Start a crawl from one URL (Zenrows Crawl API). Crawl follows the links in each page's HTML up to depth hops (no depth limit when depth is omitted), keeps the URLs that match include_patterns / exclude_patterns, and stays on the start URL's registrable domain (subdomains count).
Use crawl when you have one start page (a listing, category, blog index) and need the URLs behind it, optionally with each page's HTML. For a single page, use scrape.
Each page fetched is billed as one scrape on this account; max_pages bounds the cost (default 100), max_items bounds how many URLs are kept (default 100). Without depth, the crawl keeps following links until max_pages or max_items stops it or nothing is left to open, so set max_pages to what you are willing to spend. The start page itself is never a result, but its fetch is billed: a crawl whose include_patterns / exclude_patterns match nothing still costs that one scrape. A crawl whose pages held nothing new to follow (no link, only links to other sites, or only links the patterns reject) ends completed with no results and no error. When patterns are set, check them against the URLs the start page links to. Set output_format "html" or "markdown" to also store each kept URL's page, then read it with crawl_content.
Returns crawl_id, status and the crawl (with coverage) at once; the crawl runs asynchronously. Call crawl_wait, then crawl_results. Set idempotency_key to make a retry safe: the same key returns the crawl the first request created instead of starting another. CRAWL_NOT_ENABLED: Crawl is not enabled for this account; do not retry. CRAWL_TOO_MANY_CRAWLS: the account has reached its limit of active jobs (3 by default), shared with its Batch jobs; retry after retry_after seconds.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Start URL: a public http(s) page, such as a listing or category page | |
| depth | No | Link hops to follow from the start URL. 1 returns the start page's links; 2 also opens those. Omit for no depth limit: the crawl follows links until max_pages or max_items stops it. | |
| discovery | No | How the crawl finds the URLs it follows on each page it walks. ['links'] follows the links in each page's HTML, and is what a crawl runs with when you omit this. | |
| max_items | No | Stop once this many URLs are kept (default 100) | |
| max_pages | No | Stop once this many pages are fetched, each billed as one scrape (default 100). With output_format, every kept URL's page counts too. | |
| output_format | No | 'html' or 'markdown' also stores each kept URL's page for crawl_content, as fetched or converted to Markdown. Omit for URLs only. | |
| idempotency_key | No | Sent as the Idempotency-Key header. Reusing a key returns the crawl it created; reusing it with a different body is CRAWL_INVALID_REQUEST. | |
| exclude_patterns | No | Drop URLs containing any of these substrings, even if they match include_patterns | |
| include_patterns | No | Keep only URLs containing at least one of these substrings, e.g. ["/product/"] |