crawl_website
Recursively crawl an entire website or documentation section with depth limits, URL pattern filtering, and token accounting.
Instructions
Recursively crawl an entire website or documentation section with depth limits, URL pattern filtering, and token accounting.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"start_url": "<start_url>"
},
"name": "crawl_website"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| mode | No | Crawl mode ('fast' native fetch or 'deep' Playwright browser). | fast |
| max_depth | No | Maximum link traversal depth (default: 2). | |
| max_pages | No | Maximum total pages to crawl (default: 5, max: 20). | |
| start_url | Yes | Root URL to start the recursive crawl from. | |
| image_mode | No | Formatting of images. | ignore |
| link_format | No | Formatting of extracted hyperlinks. | inline |
| exclude_patterns | No | Substring patterns (`*` = wildcard) for URLs to ignore (e.g. ['/login', '/tags/']). Max 10. | |
| include_patterns | No | Substring patterns (`*` = wildcard) that candidate URLs must match (e.g. ['/docs/']). Max 10. |