security_robots_txt_generator
Generate RFC 9309-compliant robots.txt files from structured rule groups. Parse existing files or choose from preset rule sets.
Instructions
Robots.txt Generator (Build, Parse, and Preset RFC 9309 Crawler Rules). Assemble, parse, or template an RFC 9309 robots.txt file from structured user-agent groups. The 'operation' field selects the mode: 'generate' serializes "input" (groups of user-agents with allow/disallow paths, optional crawl-delay, sitemaps, host, comment) into robots.txt text plus validation warnings; 'parse' round-trips an existing robots.txt string back into the same structured shape; 'presets' returns 12 ready-made rule sets (allow-all, block-all, WordPress, Drupal, Joomla, Magento, Shopify, Ghost, Next.js, block-AI-scrapers, custom); 'commonUserAgents' returns a 50-entry crawler reference table. Generation only assembles and validates strings — it does NOT fetch, test, or deploy the file, and crawler compliance is voluntary. Use seo_sitemap_generator to build the sitemap this file points at, or security_htaccess_generator for Apache server-config directives rather than crawler access rules. Pure local computation: read-only, non-de
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| operation | Yes | Mode to run. 'generate' needs "input"; 'parse' needs "text"; 'presets' and 'commonUserAgents' take no other fields. | |
| input | No | Robots.txt definition for operation "generate". Requires a non-empty "groups" array. | |
| text | No | Existing robots.txt text to parse for operation "parse". |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| success | No | True when the operation succeeded. | |
| operation | No | Echo of the operation performed. | |
| result | No | Payload for the chosen operation. For "generate": robotsTxt/warnings/lineCount. For "parse": groups/sitemaps/host/comment. For "presets" and "commonUserAgents" this is instead a JSON array. |