Skip to main content
Glama

Stormap

Generate robots.txt rules for AI crawlers

generate_ai_robots_txt
Read-only

Generate robots.txt rules for AI crawlers from a policy: "open" (allow all), "balanced" (allow search and user-triggered fetchers, block bulk training crawlers) or "block" (block all). Optional per-bot overrides, allowed/blocked paths, sitemap and crawl delay. No network access.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
policyNobalanced
site_urlNo
overridesNoe.g. {"GPTBot":"allow","CCBot":"block"}
allow_pathsNo
crawl_delayNo
sitemap_urlNo
disallow_pathsNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and openWorldHint=false; the description reinforces this with 'No network access,' which is genuinely useful for an agent reasoning about side effects. It does not, however, disclose the output shape or whether generated rules are written anywhere.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with purpose and then the parameter inventory; every clause maps to real behavior. The dense parenthetical list of options is efficient rather than padded, though it reads as a run-on.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With 7 parameters, no required fields, no output schema and near-zero schema coverage, the description covers the option space but omits what the tool returns (raw robots.txt text vs. a file), how site_url interacts with the generated rules, and whether output is persisted. Adequate but with visible gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is only 14% (just the `overrides` example), so the description carries most of the burden and does so well: it explains the policy enum values, per-bot overrides, allow/disallow paths, sitemap and crawl delay. It leaves `site_url` and path formatting unaddressed, so it is not fully compensating.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description gives a specific verb and resource ('Generate robots.txt rules for AI crawlers') and scopes it by input policy. It is clear but never names or differentiates itself from the sibling generate_llms_txt, so an agent must infer which generator to pick.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The policy definitions (open/balanced/block) implicitly tell an agent which mode suits which intent, but there is no explicit when-to-use guidance, no prerequisites, and no statement of when to prefer this over generate_llms_txt or check_ai_crawler_access.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources