robots.txt generator for AI crawlers
robots_txt_generatorGenerate a robots.txt file to allow or block AI crawlers by purpose—training, search, and user-triggered fetches—plus private paths and a sitemap, without blocking Googlebot.
Instructions
Writes a robots.txt file that allows or blocks AI crawlers by purpose: model training (GPTBot, ClaudeBot, Google-Extended), AI search (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-triggered fetches (ChatGPT-User, Claude-User, Perplexity-User), plus paths to keep private and a sitemap. Never blocks Googlebot.
Tool call template
This tool declares no required parameters. Add optional parameters from the input schema as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {},
"name": "robots_txt_generator"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| ai_search | No | Whether AI search crawlers may crawl the site. | allow |
| ai_training | No | Whether AI model training crawlers may crawl the site. | allow |
| sitemap_url | No | Optional full sitemap URL, for example https://example.com/sitemap.xml. | |
| ai_user_fetch | No | Whether assistants may fetch pages when a user asks about them. | allow |
| disallow_paths | No | Optional paths to keep out of all crawlers, each starting with /, for example /wp-admin/. |