Generate robots.txt rules for AI crawlers
generate_ai_robots_txtGenerate robots.txt rules for AI crawlers from a policy: "open" (allow all), "balanced" (allow search and user-triggered fetchers, block bulk training crawlers) or "block" (block all). Optional per-bot overrides, allowed/blocked paths, sitemap and crawl delay. No network access.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| policy | No | balanced | |
| site_url | No | ||
| overrides | No | e.g. {"GPTBot":"allow","CCBot":"block"} | |
| allow_paths | No | ||
| crawl_delay | No | ||
| sitemap_url | No | ||
| disallow_paths | No |