crawl_site
Crawls a website and returns structured JSON with SEO, performance, and security metrics for comprehensive site audits.
Instructions
Run a full site crawl with SiteOne Crawler. Returns structured JSON with all crawled URLs and their SEO, performance, and security metrics. Use for comprehensive site audits.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Target URL to crawl (must include protocol) | |
| device | No | Device user agent | desktop |
| timeout | No | Per-request timeout in seconds | |
| workers | No | Concurrent workers | |
| max_depth | No | Maximum crawl depth (SiteOne default is unlimited; 10 is safer for MCP use) | |
| extra_columns | No | Additional columns via XPath or regex | |
| disable_images | No | Skip image resources | |
| disable_styles | No | Skip CSS resources | |
| max_reqs_per_sec | No | Max requests per second | |
| disable_javascript | No | Skip JavaScript resources |