Crawl and audit a public site
edgeo_crawl_siteCrawl a public site within same-host limits while respecting robots.txt and crawl-delay, using the start URL and same-host sitemap URLs to collect HTML pages for auditing.
Instructions
Run a bounded same-host HTML crawl seeded from the start URL and same-host sitemap URLs. Reads robots.txt first, applies its rules and crawl-delay, and never renders JavaScript or follows links to other hosts.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Public HTTP or HTTPS start URL. | |
| delayMs | No | ||
| maxPages | No |