ai_crawler_policy_check
Check a site's robots.txt for explicit directives targeting known AI crawlers (GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended, Bytespider, Amazonbot, and others used for LLM training or AI search/answer products), and check for an llms.txt file. Useful for publishers deciding whether their content policy toward AI crawlers matches their intent, or for auditing a competitor's stance.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Any URL on the site to check (origin is derived from it) |