AIReady

Generate robots.txt for AI

robots_txt_for_ai

Read-onlyIdempotent

Generate robots.txt User-agent rules for all known AI crawlers from a chosen policy (max_visibility, search_only, block_training, block_all_ai), with optional Content-Signal and Sitemap lines, to paste into an existing robots.txt. Use when writing AI crawler rules. Do NOT use to check a live site's current rules (use check_ai_readiness) or for bot background (use crawler_policy_guide).

Input Schema

Name	Required	Description
`policy`	Yes	AI policy preset: max_visibility = allow every AI bot; search_only or block_training = allow search/answer bots, block training crawlers (same output); block_all_ai = disallow every known AI bot.
`sitemap_url`	No	Optional absolute sitemap URL appended as a Sitemap: line, e.g. https://example.com/sitemap.xml.
`content_signals`	No	Append a Content-Signal line (search / ai-input / ai-train) matching the policy. Default true; pass false to omit.

Output Schema

Name	Required	Description
`notes`	Yes	Guidance on merging and interpreting the rules.
`policy`	Yes	The policy preset the rules were generated for.
`disclaimer`	Yes	Informational disclaimer: crawler behaviour can change, verify against provider docs.
`robots_txt`	Yes	Generated robots.txt text block, newline-terminated.

Schema Changelog

Changes observed during successful MCP inspections.

Changed5 schema fields changedOct 24, 2026

removedInput schema / additionalProperties
```
Removed value: -false
```

changedInput schema / properties / content_signals / description

Previous value: -"Add Content-Signal lines (default true)"New value: +"Append a Content-Signal line (search / ai-input / ai-train) matching the policy. Default true; pass false to omit."

changedInput schema / properties / policy / description

Previous value: -"AI crawler policy"New value: +"AI policy preset: max_visibility = allow every AI bot; search_only or block_training = allow search/answer bots, block training crawlers (same output); block_all_ai = disallow every known AI bot."

changedInput schema / properties / sitemap_url / description

Previous value: -"Optional sitemap URL"New value: +"Optional absolute sitemap URL appended as a Sitemap: line, e.g. https://example.com/sitemap.xml."

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "properties": {
+    "disclaimer": {
+      "description": "Informational disclaimer: crawler behaviour can change, verify against provider docs.",
+      "type": "string"
+    },
+    "notes": {
+      "description": "Guidance on merging and interpreting the rules.",
+      "items": {
+        "type": "string"
+      },
+      "type": "array"
+    },
+    "policy": {
+      "description": "The policy preset the rules were generated for.",
+      "enum": [
+        "max_visibility",
+        "search_only",
+        "block_training",
+        "block_all_ai"
+      ],
+      "type": "string"
+    },
+    "robots_txt": {
+      "description": "Generated robots.txt text block, newline-terminated.",
+      "type": "string"
+    }
+  },
+  "required": [
+    "policy",
+    "robots_txt",
+    "notes",
+    "disclaimer"
+  ],
+  "type": "object"
+}

Changed5 schema fields changedOct 19, 2026

addedInput schema / additionalProperties
```
Added value: +false
```

changedInput schema / properties / content_signals / description

Previous value: -"Append a Content-Signal line (search / ai-input / ai-train) matching the policy. Default true; pass false to omit."New value: +"Add Content-Signal lines (default true)"

changedInput schema / properties / policy / description

Previous value: -"AI policy preset: max_visibility = allow every AI bot; search_only or block_training = allow search/answer bots, block training crawlers (same output); block_all_ai = disallow every known AI bot."New value: +"AI crawler policy"

changedInput schema / properties / sitemap_url / description

Previous value: -"Optional absolute sitemap URL appended as a Sitemap: line, e.g. https://example.com/sitemap.xml."New value: +"Optional sitemap URL"

changedOutput schema / (root)

Previous value: -{
-  "properties": {
-    "disclaimer": {
-      "description": "Informational disclaimer: crawler behaviour can change, verify against provider docs.",
-      "type": "string"
-    },
-    "notes": {
-      "description": "Guidance on merging and interpreting the rules.",
-      "items": {
-        "type": "string"
-      },
-      "type": "array"
-    },
-    "policy": {
-      "description": "The policy preset the rules were generated for.",
-      "enum": [
-        "max_visibility",
-        "search_only",
-        "block_training",
-        "block_all_ai"
-      ],
-      "type": "string"
-    },
-    "robots_txt": {
-      "description": "Generated robots.txt text block, newline-terminated.",
-      "type": "string"
-    }
-  },
-  "required": [
-    "policy",
-    "robots_txt",
-    "notes",
-    "disclaimer"
-  ],
-  "type": "object"
-}New value: +null

Changed4 schema fields changedOct 10, 2026

changedInput schema / properties / content_signals / description

Previous value: -"True if content signals should be included, False otherwise"New value: +"Append a Content-Signal line (search / ai-input / ai-train) matching the policy. Default true; pass false to omit."

changedInput schema / properties / policy / description

Previous value: -"AI policy rules as a string, e.g. 'Disallow: /private/'"New value: +"AI policy preset: max_visibility = allow every AI bot; search_only or block_training = allow search/answer bots, block training crawlers (same output); block_all_ai = disallow every known AI bot."

changedInput schema / properties / sitemap_url / description

Previous value: -"URL of the sitemap, e.g. 'https://example.com/sitemap.xml'"New value: +"Optional absolute sitemap URL appended as a Sitemap: line, e.g. https://example.com/sitemap.xml."

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "properties": {
+    "disclaimer": {
+      "description": "Informational disclaimer: crawler behaviour can change, verify against provider docs.",
+      "type": "string"
+    },
+    "notes": {
+      "description": "Guidance on merging and interpreting the rules.",
+      "items": {
+        "type": "string"
+      },
+      "type": "array"
+    },
+    "policy": {
+      "description": "The policy preset the rules were generated for.",
+      "enum": [
+        "max_visibility",
+        "search_only",
+        "block_training",
+        "block_all_ai"
+      ],
+      "type": "string"
+    },
+    "robots_txt": {
+      "description": "Generated robots.txt text block, newline-terminated.",
+      "type": "string"
+    }
+  },
+  "required": [
+    "policy",
+    "robots_txt",
+    "notes",
+    "disclaimer"
+  ],
+  "type": "object"
+}

Changed1 schema field changedOct 10, 2026

changedOutput schema / (root)

Previous value: -{
-  "properties": {
-    "content_signal_lines": {
-      "description": "Optional Content-Signal lines included in the robots.txt if requested.",
-      "items": {
-        "type": "string"
-      },
-      "type": "array"
-    },
-    "robots_txt_content": {
-      "description": "The generated robots.txt content based on the chosen AI policy.",
-      "type": "string"
-    },
-    "sitemap_reference": {
-      "description": "The sitemap URL reference included in the robots.txt if provided.",
-      "type": "string"
-    }
-  },
-  "required": [
-    "robots_txt_content"
-  ],
-  "type": "object"
-}New value: +null

Changed3 schema fields changedOct 6, 2026

addedInput schema / properties / content_signals / description

Added value: +"True if content signals should be included, False otherwise"

addedInput schema / properties / policy / description

Added value: +"AI policy rules as a string, e.g. 'Disallow: /private/'"

addedInput schema / properties / sitemap_url / description

Added value: +"URL of the sitemap, e.g. 'https://example.com/sitemap.xml'"

Changed1 schema field changedOct 6, 2026

changedOutput schema / (root)

Previous value: -nullNew value: +{
+  "properties": {
+    "content_signal_lines": {
+      "description": "Optional Content-Signal lines included in the robots.txt if requested.",
+      "items": {
+        "type": "string"
+      },
+      "type": "array"
+    },
+    "robots_txt_content": {
+      "description": "The generated robots.txt content based on the chosen AI policy.",
+      "type": "string"
+    },
+    "sitemap_reference": {
+      "description": "The sitemap URL reference included in the robots.txt if provided.",
+      "type": "string"
+    }
+  },
+  "required": [
+    "robots_txt_content"
+  ],
+  "type": "object"
+}

First observedOct 4, 2026

TDQS

A4.5/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish readOnlyHint=true, idempotentHint=true, destructiveHint=false, so the safety profile is covered. The description adds genuine context beyond them: the output is a snippet to paste into an existing robots.txt rather than something applied to a site. It does not note that search_only and block_training return identical output (that detail lives only in the schema).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences: purpose/scope first, then when-to-use, then exclusions with alternatives. Nothing is repeated from the schema and every clause carries routing or output information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Output schema exists, so return values need no explanation, and annotations carry the safety profile. For a single-required-param generator, the description supplies everything an agent needs: what is produced, where it is meant to be used, and which sibling to pick instead.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents policy semantics, sitemap_url format, and the content_signals default. The description names the four presets and mentions the optional Content-Signal and Sitemap lines but adds no syntax or format detail beyond the schema. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and artifact ('Generate robots.txt User-agent rules for all known AI crawlers') plus the four policy presets, so the agent knows exactly what comes back: a rule snippet. It is clearly distinguishable from siblings like llms_txt_draft and schema_jsonld_sample, which produce different artifacts.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit when-to-use ('Use when writing AI crawler rules') and two explicit when-not-to-use cases with named alternatives (check_ai_readiness for live site checks, crawler_policy_guide for bot background). Routing is unambiguous with no inference required.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

All AIReady tools Inspect

Try in Browser

Give agents eyes on the webHosted screenshot MCP. Send a URL, get a pixel-perfect image. $0.002 per grab. Works with Claude, Cursor, and Codex.grabbit.liveAd

Glama MCP Gateway

Add one secure layer between your agents and this server.

Generate robots.txt for AI

Input Schema

Output Schema

Schema Changelog

TDQS

Glama MCP Gateway

Resources