Generate an AI-crawler robots.txt
generate_robots_txtRead-onlyIdempotent
Builds a robots.txt file that explicitly allows or blocks named AI crawlers from Citehound’s tracked list of 10, plus an optional Sitemap line. Pure text generation — fetches nothing. Call list_ai_crawlers first if you need the exact tracked crawler names.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| allow | No | Crawler names to explicitly allow, e.g. ["GPTBot", "ClaudeBot"]. Must be names from the tracked list. | |
| block | No | Crawler names to explicitly block. Must be names from the tracked list, and must not overlap with "allow". | |
| sitemap_url | No | Optional absolute sitemap URL to add as a Sitemap: line, e.g. "https://example.com/sitemap.xml". |