Crawl and audit a public site
edgeo_crawl_siteCrawl a bounded sample of same-host HTML pages from a start URL and sitemap, respecting robots.txt and crawl-delay, to audit public site content without rendering JavaScript or leaving the host.
Instructions
Run a bounded same-host HTML crawl seeded from the start URL and same-host sitemap URLs. Defaults to 50 attempted URLs and accepts up to 200. Preserves trailing slashes when requesting URLs, reads robots.txt first, applies its rules and crawl-delay, separates non-HTML sitemap resources, and never renders JavaScript or follows links to other hosts. Reports known queued URL counts, not sitewide coverage.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "edgeo_crawl_site"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Public HTTP or HTTPS start URL. | |
| delayMs | No | ||
| maxPages | No | Maximum URL fetch attempts, including non-HTML sitemap resources and URL variants that redirect to an already seen page. |