Skip to main content
Glama

robots.txt generator

robots_txt_generator
Read-onlyIdempotent

Builds a robots.txt file: allow everything, block everything or custom Allow and Disallow paths, with sitemap URLs, a crawl delay and switches that block AI crawlers such as GPTBot and ClaudeBot. Returns the file and findings.

Tool call template

This tool declares no required parameters. Add optional parameters from the input schema as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {},
    "name": "robots_txt_generator"
  }
}

Input Schema

NameRequiredDescriptionDefault
modeNoallow lets every crawler in (default), block shuts them all out, custom uses disallow and allow paths.
allowNoPaths to allow in custom mode, each starting with /.
disallowNoPaths to block in custom mode, each starting with /.
sitemapsNoFull sitemap URLs.
crawl_delayNoCrawl-delay in seconds.
block_ai_botsNoAI crawlers to block completely. One of: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, CCBot, Applebot-Extended, Bytespider.

Schema Changelog

Changes observed during successful MCP inspections.

  1. Added

TDQS

B3.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false and openWorldHint=false, so safety and mutation behavior are covered structurally. The description contributes real value by stating "Returns the file and findings" in the absence of an output schema, and by hinting at the AI-crawler blocking behavior. It never clarifies whether anything is written to a server or whether the result is text versus a downloadable artifact, so with annotations doing most of the work a 3 is fair.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single dense sentence that front-loads the core action and then layers on capabilities, with no filler or repetition. The enumeration of bot names is slightly redundant with the schema enum, but it reads as a compact capability summary rather than padding.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With six all-optional parameters, full schema coverage and no output schema, the description covers enough to invoke the tool, and "Returns the file and findings" partially covers the missing return contract. What it lacks is context an agent would want for a generator sitting next to a tester: whether the output is a preview or a persisted file, and when to prefer this over robots_txt_test.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% for all six parameters, including the mode enum explanation, path format rules, sitemap URL expectations, crawl-delay units and the enumerated AI crawler list. The description restates these at a high level rather than adding new constraints (e.g., mutual exclusivity of allow/disallow in custom mode), so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource ("Builds a robots.txt file") and enumerates the three generation modes plus the optional features it supports, so an agent knows exactly what artifact comes out. It does not name the closest sibling (robots_txt_test) or state how generating differs from testing, so it stays below a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no when-to-use / when-not-to-use guidance and no routing to alternatives such as robots_txt_test for validating an existing file. The only usage-flavored content ("allow everything, block everything or custom") merely paraphrases the mode enum already documented in the schema, so it adds no selection guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Try in Browser

Glama MCP Gateway

Add one secure layer between your agents and this server.

Resources