Write robots.txt rules for AI crawlers
generate_ai_robots_rulesWrite robots.txt groups for every AI crawler from a simple policy: allow or block search crawlers, crawlers that fetch pages when a user asks, and training crawlers, keeping private paths private. Use this when someone wants to allow AI search but block AI training (or any other mix), or asks for an AI-ready robots.txt. Returns the text to paste plus notes on crawlers that may ignore robots.txt.
Tool call template
This tool declares no required parameters. Add optional parameters from the input schema as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {},
"name": "generate_ai_robots_rules"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| sitemap_url | No | The sitemap URL to list | |
| allow_search | No | Allow crawlers that build AI search indexes (OAI-SearchBot, Claude-SearchBot, PerplexityBot…) | |
| private_paths | No | Paths no crawler should read, for example /admin or /checkout | |
| allow_training | No | Allow crawlers that collect pages for model training (GPTBot, ClaudeBot, CCBot…) | |
| allow_user_requests | No | Allow crawlers that fetch a page because a user asked (ChatGPT-User, Claude-User, Perplexity-User…) |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| status | Yes |