crawl_site
Crawl up to 10 same-origin pages while respecting robots.txt, then detect duplicate titles and meta descriptions to support SEO audits.
Instructions
Crawl up to 10 pages on the exact same origin, respecting robots.txt. Detect duplicate titles and descriptions.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | ||
| max_pages | No |