util_robots_inspect
Inspect a domain's robots.txt and sitemap to identify crawling restrictions and sitemap URLs before indexing.
Instructions
Read robots.txt rules and sitemap declarations before crawling or indexing a domain.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| arguments | Yes |