Generate robots.txt
generate_robots_txtGenerate a ready-to-save robots.txt for any website, merging existing rules with AI-crawler policies. Choose a preset (allow-all, search-only, recommended, or block-all) to control AI access.
Instructions
Generate a complete, ready-to-save AI-crawler-aware robots.txt for a website. Fetches the existing robots.txt (if any) and returns the finished file in generated.robotsTxt — the existing rules with an AI-crawler policy section merged in — plus the per-bot allow/block breakdown. Presets: 'allow-all' (public businesses), 'search-only' (allow AI search, block training), 'recommended' (allow major AI assistants that cite sources, block training-only bots), 'block-all'. Write generated.robotsTxt to the site's /robots.txt. Requires an API key.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The website to generate a robots.txt for. | |
| preset | No | The AI-crawler policy preset to apply. | recommended |