crawl_site
Crawl a full website starting from a seed URL with configurable depth and page limits. Output clean markdown or structured JSON for further analysis.
Instructions
Crawl an entire website starting from a seed URL using DFS strategy.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The seed URL to start crawling from | |
| timeout | No | Overall site crawl timeout in seconds (default: 120, must be >= 1) | |
| max_depth | No | Maximum depth to crawl (default: 2, 0 = seed page only) | |
| max_pages | No | Maximum number of pages to crawl (default: 25) | |
| dedup_mode | No | Markdown dedup mode - "exact" (default) or "off" | exact |
| remove_links | No | Remove all links from the markdown output (default: false) | |
| output_format | No | Output format - "markdown" (default) or "json" - markdown: Clean concatenated markdown with URL headers and timestamps - json: Full JSON with metadata, references, and crawl statistics | markdown |
| storage_state | No | Path to Playwright storage_state JSON for authenticated crawling | |
| include_subdomains | No | Whether to include subdomains in the crawl (default: false) |