generate_robots_txt
Generate a robots.txt file with customizable allow/disallow paths, crawl delay, and sitemap. Optionally block major AI training crawlers to manage bot access.
Instructions
Generate a robots.txt. Optionally block the major AI training crawlers.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| allow_paths | No | ||
| crawl_delay | No | ||
| sitemap_url | No | ||
| disallow_paths | No | ||
| block_ai_crawlers | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |