Draft robots.txt rules for AI crawlers
draft_robots_txt_for_ai_crawlersDrafts robots.txt groups for the AI crawlers Agent Ready tracks, and a Content-Signal line, from what the site owner wants: crawlers that index for AI search, crawlers that fetch a page because a user asked, and crawlers that collect training data, each allowed or blocked, with per-crawler overrides. It fetches nothing. Use it after check_robots_sitemap_llms_txt or check_ai_crawler_access shows crawlers blocked or allowed against the owner's wishes. result has robotsTxt (the lines to merge), contentSignal, decisions (each crawler with its role and choice) and unknownOverrides (tokens not recognised). Returns JSON: summary (one sentence) and result (the details below); resultId and reportUrl are null, as drafts are not saved. Errors come back as one plain sentence.
Tool call template
This tool declares no required parameters. Add optional parameters from the input schema as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {},
"name": "draft_robots_txt_for_ai_crawlers"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| search | No | Whether AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) may read the site: allow or block. Defaults to allow. | |
| training | No | Whether training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) may read it: allow or block. Defaults to allow. | |
| overrides | No | Per-crawler choices that win over the role choices, by robots.txt token, such as {"GPTBot": "block"}. Defaults to none. | |
| userFetch | No | Whether crawlers fetching for a user (ChatGPT-User, Claude-User) may read it: allow or block. Defaults to allow. |