Crawl a website
deepcrawlRead multiple pages from one website in a single task by crawling sitemap URLs that match path rules, returning citable titles, URLs, and Markdown content for site-wide research.
Instructions
Read many pages of one website in a single task, such as a documentation section, when the user needs content from across a site rather than one page. Finds the site's URLs through its sitemap, keeps those whose path matches the path rules, and returns up to max_pages pages as citable results with title, URL, and Markdown text, plus page counts and the URLs that failed. Costs 20 credits per started task, however many pages it reads. Waits about 45 seconds; if the crawl is still running, the result holds a task_id, and calling deepcrawl again with that task_id collects the pages at no cost. Use crawl for a single page and sitemap for a list of URLs without their content.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | Site or section to crawl, such as https://example.com or https://example.com/docs/; required unless task_id is given | |
| type | No | Where URLs come from: sitemap uses the site's sitemap, or the links on the start page when it has none; all uses every link discovered from the start URL, which can include other sites | sitemap |
| task_id | No | task_id from an earlier deepcrawl result that was still running; collects that crawl for free instead of starting a new one, and the other arguments are ignored | |
| max_pages | No | Most pages to read, taken in discovery order after the path rules; the price is the same for any number | |
| exclude_paths | No | Skip URLs whose path matches one of these rules, with the same syntax as include_paths; exclusion wins over inclusion | |
| include_paths | No | Only read URLs whose path matches one of these rules. A rule without * is a path prefix on segment boundaries ("/docs" matches /docs/api but not /docsearch); a rule with * must match the whole path, and * also matches / ("/blog/*") |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| note | No | What to do next: call again with task_id, or where to find text that was cut | |
| stats | No | Page counts: links discovered, links matching the path rules, matches left out by max_pages, and pages crawled or failed | |
| failed | No | Pages that could not be crawled, with the reason | |
| status | Yes | completed when the crawl has finished; running when it needs more time and results is empty | |
| message | No | Summary of the finished crawl, including why nothing was crawled | |
| results | Yes | Crawled pages in discovery order; text holds each page's Markdown, shortened when metadata.truncated is true | |
| task_id | Yes | Crawl task id; pass it back as task_id while status is running | |
| truncated | No | Whether page text was shortened or left out to bound the response | |
| result_url | No | JSON file with every crawled page in full |