crawl_deep
Read-only
Crawl and extract content from multiple pages of a website to build knowledge bases, index documentation, or audit site structure.
Instructions
Use this when you need to discover and optionally extract content from many pages within a site — e.g. building a knowledge base, indexing docs, or auditing all pages. Use map_site first to estimate scope, then crawl_deep for content. Example: crawl_deep({url: "https://docs.example.com", max_depth: 3, max_pages: 200, extract_content: true})
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Starting URL for the crawl | |
| session | No | Shared cookie-jar/session for login-then-crawl workflows | |
| max_depth | No | Maximum crawl depth from starting URL | |
| max_pages | No | Maximum number of pages to crawl | |
| concurrency | No | Number of concurrent requests | |
| domain_filter | No | Per-domain allow/deny lists and crawl rules | |
| respect_robots | No | Respect robots.txt directives | |
| extract_content | No | Extract page content during crawl | |
| follow_external | No | Follow links to external domains | |
| exclude_patterns | No | URL patterns to exclude (regex) | |
| include_patterns | No | URL patterns to include (regex) | |
| content_max_length | No | Maximum characters of page content to include per page (default 500); sets a truncated flag when trimmed | |
| enable_link_analysis | No | Compute PageRank/link-graph analysis over crawled pages | |
| import_filter_config | No | JSON string of a previously exported domain-filter config | |
| link_analysis_options | No | PageRank tuning options |