Start a crawl
crawlStarts a site-section sweep from a URL, following links in the start path's subtree up to page and depth limits, and returns a task ID to track progress.
Instructions
Start crawling a section of one site: from url, follow links in the start path's subtree on its host and www twin, up to maxPages pages and maxDepth links deep. Returns { taskId } at once (with maxPages when the default of 100 applied); wait_crawl waits for it, then get_crawl_pages reads the pages. How long the server keeps them is not stated. Each page takes scrape's page options (formats, onlyMainContent, includeTags, waitFor, timeout, verify and the rest; their descriptions are on scrape). Similar URLs fold into one page, and every collapsed or refused link is counted in get_crawl's discovery.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| mode | No | standard (default) or research; for signed-in pages send a batch in mode authed. | |
| access | No | standard (default) or enhanced: also the paid providers the operator approved. | |
| maxAge | No | ||
| minAge | No | ||
| mobile | No | ||
| verify | No | Same as scrape. | |
| formats | No | Same as scrape. | |
| headers | No | ||
| sitemap | No | include (default): also queue the site's sitemap URLs. skip: no sitemap. only: the start URL and the sitemap entries. | |
| timeout | No | ||
| waitFor | No | ||
| webhook | No | Where the job posts events, retried: a URL or { url, headers?, metadata?, events?, secretEnv? } (a W2L_WEBHOOK_SECRET_* variable signing them). | |
| blockAds | No | ||
| fastMode | No | ||
| lockdown | No | ||
| maxDepth | No | Links followed from the start URL at most. Omitted or null: no limit. | |
| maxPages | No | Pages fetched at most. Omitted: 100. null: the server's default (none on a local server; hosted servers cap it). | |
| useCached | No | ||
| excludeTags | No | ||
| includeTags | No | ||
| integration | No | Your workflow label (printable, no spaces); recorded, not sent to sites. | |
| excludePaths | No | Pathname regexes that skip a discovered link; they win over includePaths. | |
| includeLinks | No | ||
| includePaths | No | Pathname regexes a discovered link must match; the start URL is always fetched. | |
| maxFileBytes | No | ||
| storeInCache | No | ||
| idempotencyKey | No | Makes a retry safe: the same key and arguments return the first taskId. Kept 24 hours. | |
| maxConcurrency | No | Pages fetched at once, at most (only lowers the worker count). | |
| regexOnFullURL | No | Match includePaths and excludePaths against each link's canonical URL instead of its pathname. Default false. | |
| allowSubdomains | No | Follow links to every host under the start URL's apex. Default false. | |
| ignoreRobotsTxt | No | Also fetch what robots.txt disallows (recorded, with a warning). Default false. Local server only. | |
| onlyMainContent | No | ||
| crawlEntireDomain | No | Follow links anywhere on the start host, not only in the start path's subtree. Default false. | |
| allowExternalLinks | No | Follow links to any host, each with its own robots.txt; maxDepth and maxPages bound the walk. Default false. Not with allowlistedDomains. | |
| allowlistedDomains | No | ||
| removeBase64Images | No | ||
| skipTlsVerification | No | ||
| ignoreQueryParameters | No | Treat URLs that differ only in their query as one page. Default false. | |
| deduplicateSimilarURLs | No | Treat /a and /a/, www and apex, http and https as one page. Default true. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| taskId | Yes | ||
| replayed | No |