ai_crawler_audit
Audit which AI crawlers a site allows or blocks in robots.txt — sorted by CONSEQUENCE: blocking a VISIBILITY crawler (OAI-SearchBot, PerplexityBot, ChatGPT-User, Claude-User…) removes the site from live AI answers, while blocking a TRAINING crawler (GPTBot, CCBot, Google-Extended…) only opts out of model training. Those need opposite decisions. Checks 25 known AI crawlers + whether /llms.txt exists, plus general robots.txt hygiene (Sitemap directives, whole-site Disallow foot-guns). Free (no LLM calls).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| domain | Yes | Root domain to audit, e.g. 'yoursite.com'. |