spider_crawl
Crawl entire websites with configurable depth, page limits, filters, and options for JavaScript rendering, subdomains, and sitemaps. Use it to map or extract site content at scale.
Instructions
Crawl an entire website with configurable depth, limits, and filtering options
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| js | No | ||
| url | Yes | ||
| depth | No | ||
| limit | No | ||
| budget | No | ||
| headers | No | ||
| sitemap | No | ||
| stealth | No | ||
| anti_bot | No | ||
| metadata | No | ||
| blacklist | No | ||
| whitelist | No | ||
| store_data | No | ||
| subdomains | No | ||
| proxy_country | No | ||
| proxy_enabled | No | ||
| return_format | No | ||
| respect_robots | No |