Check AI crawler access & indexability
check_ai_crawlersCheck effective robots.txt status for major AI crawlers, detect blanket disallow, verify llms.txt, and flag noindex/nofollow in headers and meta tags.
Instructions
Report the effective robots.txt allow/block status for GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, and the * default (flagging blanket Disallow: / without per-bot overrides). Checks for /llms.txt, and flags noindex/nofollow in both the X-Robots-Tag response header and — critical failures invisible to a plain robots.txt check.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | Page URL to check (robots.txt/llms.txt are checked at its origin) |