crawl_deep
Read-only
Crawl a website by following its links to collect content from many pages, enabling knowledge base extraction, documentation indexing, and full-site audits.
Instructions
Use this to fetch many pages of one site by following links - a knowledge base, a docs index, a full-site audit. Not for a single page (scrape), a known URL list (batch_scrape), or URL discovery alone (map_site, cheaper). Cost: 4 credits base, grows with page count. Example: crawl_deep({url: "https://docs.example.com", max_depth: 3, max_pages: 200, extract_content: true})
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Starting URL for the crawl | |
| session | No | Shared cookie-jar/session for login-then-crawl workflows | |
| max_depth | No | Maximum crawl depth from starting URL | |
| max_pages | No | Maximum number of pages to crawl | |
| redact_pii | No | Redact personal data from the text this call returns, before it reaches your context window. true means the free regex pass over EMAIL, PHONE, FINANCIAL and SECRET. The result carries redaction:{entities,count}. Default: off | |
| concurrency | No | Number of concurrent requests | |
| domain_filter | No | Per-domain allow/deny lists and crawl rules | |
| respect_robots | No | Respect robots.txt directives | |
| extract_content | No | Extract page content during crawl | |
| follow_external | No | Follow links to external domains | |
| exclude_patterns | No | URL patterns to exclude (regex) | |
| include_patterns | No | URL patterns to include (regex) | |
| max_inline_chars | No | Largest result to return inline, in characters of its JSON. Over it, the call returns a preview plus a result_handle for read_result instead of the whole result (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS) | |
| content_max_length | No | Maximum characters of page content to include per page (default 500); sets a truncated flag when trimmed | |
| enable_link_analysis | No | Compute PageRank/link-graph analysis over crawled pages | |
| import_filter_config | No | JSON string of a previously exported domain-filter config | |
| link_analysis_options | No | PageRank tuning options |
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | No | ||
| view | No | Whether preview and read_result offsets index a text field or the pretty-printed JSON | |
| _cost | No | Cost-transparency metadata (D3.5), present when injected into the text copy of the result | |
| error | No | ||
| stats | No | ||
| cached | No | True when this response was replayed from an earlier crawl rather than crawled now; crawled_at gives its age | |
| errors | No | ||
| preview | No | The first max_inline_chars characters of the view named by view_path (or of the pretty-printed JSON) | |
| results | No | ||
| session | No | ||
| success | No | False only when the crawl was cancelled via elicitation decline | |
| redaction | No | Present when redact_pii was set: what was redacted from the text of this result | |
| truncated | No | True when the inline result is a preview | |
| view_path | No | Dotted path of the text field the view was cut from; null for the JSON view | |
| crawled_at | No | When the pages were actually fetched (ISO 8601) | |
| expires_at | No | When the stored result is dropped (ISO 8601) | |
| crawl_depth | No | ||
| duration_ms | No | ||
| error_count | No | ||
| pages_found | No | ||
| total_chars | No | Length of the full view in characters | |
| link_analysis | No | ||
| pages_crawled | No | ||
| result_handle | No | Handle for read_result; the full result is kept 1 hour | |
| site_structure | No | ||
| pages_per_second | No | ||
| domain_filter_config | No |