Write robots.txt rules for AI crawlers
generate_ai_robots_rulesRead-onlyIdempotent
Write robots.txt groups for every AI crawler from a simple policy: allow or block search crawlers, crawlers that fetch pages when a user asks, and training crawlers, keeping private paths private. Use this when someone wants to allow AI search but block AI training (or any other mix), or asks for an AI-ready robots.txt. Returns the text to paste plus notes on crawlers that may ignore robots.txt.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| sitemap_url | No | The sitemap URL to list | |
| allow_search | No | Allow crawlers that build AI search indexes (OAI-SearchBot, Claude-SearchBot, PerplexityBot…) | |
| private_paths | No | Paths no crawler should read, for example /admin or /checkout | |
| allow_training | No | Allow crawlers that collect pages for model training (GPTBot, ClaudeBot, CCBot…) | |
| allow_user_requests | No | Allow crawlers that fetch a page because a user asked (ChatGPT-User, Claude-User, Perplexity-User…) |
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| status | Yes |