robots.txt generator
robots_txt_generatorBuilds a robots.txt file: allow everything, block everything or custom Allow and Disallow paths, with sitemap URLs, a crawl delay and switches that block AI crawlers such as GPTBot and ClaudeBot. Returns the file and findings.
Tool call template
This tool declares no required parameters. Add optional parameters from the input schema as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {},
"name": "robots_txt_generator"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| mode | No | allow lets every crawler in (default), block shuts them all out, custom uses disallow and allow paths. | |
| allow | No | Paths to allow in custom mode, each starting with /. | |
| disallow | No | Paths to block in custom mode, each starting with /. | |
| sitemaps | No | Full sitemap URLs. | |
| crawl_delay | No | Crawl-delay in seconds. | |
| block_ai_bots | No | AI crawlers to block completely. One of: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, CCBot, Applebot-Extended, Bytespider. |