robots.txt generator for AI crawlers
robots_txt_generatorWrites a robots.txt file that allows or blocks AI crawlers by purpose: model training (GPTBot, ClaudeBot, Google-Extended), AI search (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-triggered fetches (ChatGPT-User, Claude-User, Perplexity-User), plus paths to keep private and a sitemap. Never blocks Googlebot.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| ai_search | No | Whether AI search crawlers may crawl the site. | allow |
| ai_training | No | Whether AI model training crawlers may crawl the site. | allow |
| sitemap_url | No | Optional full sitemap URL, for example https://example.com/sitemap.xml. | |
| ai_user_fetch | No | Whether assistants may fetch pages when a user asks about them. | allow |
| disallow_paths | No | Optional paths to keep out of all crawlers, each starting with /, for example /wp-admin/. |