Generate robots.txt
generate_robots_txtGenerates an AI-crawler-aware robots.txt for a website by merging existing rules with a selected allow, search-only, or block policy, returning a ready-to-save file.
Instructions
Generate a complete, ready-to-save AI-crawler-aware robots.txt for a website. Fetches the existing robots.txt (if any) and returns the finished file in generated.robotsTxt — the existing rules with an AI-crawler policy section merged in — plus the per-bot allow/block breakdown. Presets: 'allow-all' (every AI crawler allowed), 'recommended' (blocks only the training-only crawlers; everything that feeds AI search and answers stays allowed), 'search-only' (allows only the crawlers that feed AI search and answers, the ones the readiness score counts, and blocks the rest), 'block-all'. A file from 'recommended' or 'search-only' loses no points for crawler access. Write generated.robotsTxt to the site's /robots.txt. Requires an API key.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"url": "<url>"
},
"name": "generate_robots_txt"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| url | Yes | The website to generate a robots.txt for. | |
| preset | No | The AI-crawler policy preset to apply. | recommended |