generate_robots_txt
Create a robots.txt file that permits real-time AI citation search bots and controls offline AI model training scrapers for a specified domain.
Instructions
Generate an optimized robots.txt file that allows real-time AI citation search bots (Claude-SearchBot, OAI-SearchBot, Google-Extended) while controlling offline model training scrapers.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"domain": "<domain>"
},
"name": "generate_robots_txt"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| domain | Yes | Domain name (e.g., 'example.com') | |
| allow_training | No | Allow general AI model training web scrapers (default: false) | |
| allow_citations | No | Allow real-time AI search bots to cite content (default: true) |