Skip to main content
Glama

upGrowth Growth Toolkit

robots.txt generator for AI crawlers

robots_txt_generator
Read-onlyIdempotent

Writes a robots.txt file that allows or blocks AI crawlers by purpose: model training (GPTBot, ClaudeBot, Google-Extended), AI search (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-triggered fetches (ChatGPT-User, Claude-User, Perplexity-User), plus paths to keep private and a sitemap. Never blocks Googlebot.

Input Schema

NameRequiredDescriptionDefault
ai_searchNoWhether AI search crawlers may crawl the site.allow
ai_trainingNoWhether AI model training crawlers may crawl the site.allow
sitemap_urlNoOptional full sitemap URL, for example https://example.com/sitemap.xml.
ai_user_fetchNoWhether assistants may fetch pages when a user asks about them.allow
disallow_pathsNoOptional paths to keep out of all crawlers, each starting with /, for example /wp-admin/.

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observed

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive, so the description rightly spends its words on the crawler-to-category mapping and the hard guarantee that Googlebot is never blocked — genuine behavioral context a caller cannot infer from structured fields. Minor ambiguity: 'Writes a robots.txt file' sits in slight tension with readOnlyHint=true, since it presumably returns file content rather than persisting it.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One front-loaded sentence, no filler, with the categories separated by a clean colon-list. The crawler parentheticals make it dense, but every name earns its place by disambiguating the three enum values.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

All five parameters are optional and there is no output schema, so the description should say what comes back (raw robots.txt text? lines per bot?). It covers inputs and policy semantics well but leaves the return shape and the read-vs-write ambiguity unaddressed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so baseline is 3, but the description earns above baseline by naming which concrete crawlers each enum value governs (GPTBot vs OAI-SearchBot vs ChatGPT-User), meaning the agent understands the blast radius of each allow/block choice rather than just the field label.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('Writes a robots.txt file') and enumerates the exact scope: three crawler categories, private paths, sitemap. The generative verb also cleanly separates it from the sibling checker `ai_crawler_access_check` without needing to name it.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the description (produce a robots.txt policy) but there is no explicit when-to-use, when-not-to-use, or pointer to the access-check sibling. The one real constraint given, 'Never blocks Googlebot', is a behavioral guarantee rather than routing guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.