Crawl a site
crawl_siteCrawl a live website and set the result as the active dataset for other SEO tools, respecting robots.txt and capping at 200 pages.
Instructions
Crawl a website live (same-origin, breadth-first, robots.txt respected) and make the result the active dataset for the other tools. Capped at 200 pages; for a full-site audit with history and Search Console data, use the Crawl Cove desktop app and load its export with load_export.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "crawl_site"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Start URL, e.g. https://example.com/ | |
| maxPages | No | Page cap (default 50, max 200) | |
| ignoreRobots | No | Crawl URLs robots.txt disallows. Only for sites you own. |