Skip to main content
Glama

Generate robots.txt rules for AI crawlers

generate_ai_robots_txt
Read-only

Generate robots.txt rules for AI crawlers from a policy: "open" (allow all), "balanced" (allow search and user-triggered fetchers, block bulk training crawlers) or "block" (block all). Optional per-bot overrides, allowed/blocked paths, sitemap and crawl delay. No network access.

Input Schema

NameRequiredDescriptionDefault
policyNobalanced
site_urlNo
overridesNoe.g. {"GPTBot":"allow","CCBot":"block"}
allow_pathsNo
crawl_delayNo
sitemap_urlNo
disallow_pathsNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and openWorldHint=false, and the description reinforces this with 'No network access', usefully signaling a pure local generation. It also clarifies the blocking semantics of the 'balanced' policy, though it doesn't describe the output artifact beyond its name.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded paragraph that opens with the purpose, then the policy definitions, then the optional surface. Dense but every clause maps to a real capability; no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With 7 parameters, a nested overrides object, and no output schema, the description should do more: it omits the meaning of site_url and never describes what is returned (a robots.txt string, recommendations, etc.). The core configuration surface is covered, but gaps remain for a tool of this complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is very low (14%), so the description must carry the load, and it does explain the policy values and enumerates the optional features (per-bot overrides, allowed/blocked paths, sitemap, crawl delay) that map to the other parameters. It leaves site_url undefined, keeping it short of full compensation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (generate) and resource (robots.txt rules for AI crawlers) scoped to a named input (policy). The resource is clearly distinct from the sibling generate_llms_txt, so an agent can tell what it produces without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description defines what each policy value means (open/balanced/block), which implies the selection context, but it never states when to choose this tool over alternatives like generate_llms_txt or check_ai_crawler_access, nor any prerequisites. Usage is implied rather than guided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources