Generate a robots.txt for AI crawlers
generate_robotsBuild a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns text and changes nothing. Same engine as POST /api/generate.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| date | No | Date for the header comment (YYYY-MM-DD). Default: today (UTC). | |
| agents | No | Per-token overrides, e.g. {"GPTBot": "block"}. Tokens: letters, digits, "_", "." and "-", up to 64 characters (not "*"). At most 100. | |
| default | No | Policy for "User-agent: *" (default allow). | |
| sitemap | No | A single sitemap URL (alternative to sitemaps). | |
| sitemaps | No | Absolute http(s) sitemap URLs. At most 20. | |
| categories | No | Action per crawler category. Omitted categories get no group of their own. | |
| allow_paths | No | Paths to allow explicitly. Each path starts with "/" or "*" and has no whitespace or "#". At most 100. | |
| custom_rules | No | Extra robots.txt lines appended after validation (max 20000 bytes). | |
| disallow_paths | No | Paths to disallow for "*" and for allowed AI groups. Each path starts with "/" or "*" and has no whitespace or "#". At most 100. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| notes | Yes | ||
| summary | Yes | ||
| robots_txt | Yes | ||
| personal_agents_note | No |