spider_crawl
Crawl entire websites with configurable depth, page limits, filters, and options for JavaScript rendering, subdomains, and sitemaps. Use it to map or extract site content at scale.
Instructions
Crawl an entire website with configurable depth, limits, and filtering options
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "spider_crawl"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| js | No | ||
| url | Yes | ||
| depth | No | ||
| limit | No | ||
| budget | No | ||
| headers | No | ||
| sitemap | No | ||
| stealth | No | ||
| anti_bot | No | ||
| metadata | No | ||
| blacklist | No | ||
| whitelist | No | ||
| store_data | No | ||
| subdomains | No | ||
| proxy_country | No | ||
| proxy_enabled | No | ||
| return_format | No | ||
| respect_robots | No |