Start a crawl
crawlStart crawling a section of one site: from url, follow links in the start path's subtree on its host and www twin, up to maxPages pages and maxDepth links deep. Returns { taskId } at once (with maxPages when the default of 100 applied); wait_crawl waits for it, then get_crawl_pages reads the pages. Each page takes scrape's page options (formats, onlyMainContent, includeTags, waitFor, timeout, verify and the rest; their descriptions are on scrape). Similar URLs fold into one page, and every collapsed or refused link is counted in get_crawl's discovery. On hosted Octocrawl a job runs in slices: a crawl reads at most 500 pages and a batch 1,000 URLs, each page costs credits as a scrape does (maxCredits caps the job), and results are kept 7 days after it ends.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "crawl"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| mode | No | The declared identity the page is read under: standard (default) or research. A hosted server holds no saved logins, so it has no authed mode. | |
| mobile | No | ||
| verify | No | Same as scrape. | |
| formats | No | What to return (default markdown): markdown, links, html, rawHtml, images, tables, json, attributes ({ selectors }), list ({ itemSelector?, fields? }: a record per repeated item, detected without itemSelector), screenshot or screenshot@fullPage (browser only). On an account's credits a browser page costs 5; with fewer left, a screenshot is refused. | |
| headers | No | ||
| sitemap | No | include (default): also queue the site's sitemap URLs. skip: no sitemap. only: the start URL and the sitemap entries. | |
| timeout | No | ||
| waitFor | No | ||
| blockAds | No | ||
| fastMode | No | ||
| maxDepth | No | Links followed from the start URL at most. Omitted or null: no limit. | |
| maxPages | No | Pages fetched at most. Omitted: 100. null: the server's default (none on a local server; hosted servers cap it). | |
| maxCredits | No | The most credits this job may spend; it stops when the next page would pass it. | |
| excludeTags | No | ||
| includeTags | No | ||
| integration | No | Your workflow label; recorded, not sent to sites. | |
| excludePaths | No | Pathname regexes that skip a discovered link; they win over includePaths. | |
| includeLinks | No | ||
| includePaths | No | Pathname regexes a discovered link must match; the start URL is always fetched. | |
| maxFileBytes | No | ||
| idempotencyKey | No | Makes a retry safe: the same key and arguments return the first taskId. Kept 24 hours. | |
| maxConcurrency | No | Pages fetched at once, at most (only lowers the worker count). | |
| regexOnFullURL | No | Match includePaths and excludePaths against each link's canonical URL instead of its pathname. Default false. | |
| allowSubdomains | No | Follow links to every host under the start URL's apex. Default false. | |
| onlyMainContent | No | ||
| crawlEntireDomain | No | Follow links anywhere on the start host, not only in the start path's subtree. Default false. | |
| allowExternalLinks | No | Follow links to any host, each with its own robots.txt; maxDepth and maxPages bound the walk. Default false. Not with allowlistedDomains. | |
| allowlistedDomains | No | ||
| removeBase64Images | No | ||
| ignoreQueryParameters | No | Treat URLs that differ only in their query as one page. Default false. | |
| deduplicateSimilarURLs | No | Treat /a and /a/, www and apex, http and https as one page. Default true. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| taskId | Yes | ||
| replayed | No |