generate_robots
Create a robots.txt file to block specific AI crawlers, include sitemap URLs, and add custom rules. Control which bots are allowed or denied.
Instructions
Generate a robots.txt file with specified blocked bots, sitemap URLs, and custom rules.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| block_all_ai | No | If true, blocks all known AI crawlers (but not search engines like Googlebot) | |
| blocked_bots | No | Array of bot user-agent strings to block (e.g. ['GPTBot', 'ClaudeBot', 'CCBot']). Use list_ai_bots to see available user-agents. | |
| custom_rules | No | Additional custom robots.txt rules to append (raw robots.txt syntax) | |
| sitemap_urls | No | Array of sitemap URLs to include (e.g. ['https://example.com/sitemap.xml']) |