Skip to main content
Glama

generate_robots

Create a robots.txt file to block specific AI crawlers, include sitemap URLs, and add custom rules. Control which bots are allowed or denied.

Instructions

Generate a robots.txt file with specified blocked bots, sitemap URLs, and custom rules.

Input Schema

TableJSON Schema
NameRequiredDescriptionDefault
block_all_aiNoIf true, blocks all known AI crawlers (but not search engines like Googlebot)
blocked_botsNoArray of bot user-agent strings to block (e.g. ['GPTBot', 'ClaudeBot', 'CCBot']). Use list_ai_bots to see available user-agents.
custom_rulesNoAdditional custom robots.txt rules to append (raw robots.txt syntax)
sitemap_urlsNoArray of sitemap URLs to include (e.g. ['https://example.com/sitemap.xml'])
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description carries full burden. It only says 'Generate a robots.txt file' without disclosing side effects, authorization needs, or output behavior (e.g., file creation vs. returning content).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence listing key elements. It is front-loaded and efficient, though it could benefit from structured formatting for clarity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a generation tool with 4 optional parameters and no output schema, the description lacks critical details: default behavior when no parameters provided, output format (file vs. string), and prerequisites (e.g., domain context).

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, but the description adds meaningful context: explains block_all_ai excludes Googlebot, references list_ai_bots for blocked_bots, and clarifies custom_rules uses raw syntax. This goes beyond names and schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool generates a robots.txt file with specified blocked bots, sitemaps, and custom rules. It distinguishes from siblings like analyze_robots and list_ai_bots which have different purposes.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no guidance on when to use this tool versus siblings. It only briefly mentions using list_ai_bots for available user-agents within a single parameter description, but lacks explicit when-to-use or when-not-to-use context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Install Server

Other Tools

Latest Blog Posts

MCP directory API

We provide all the information about MCP servers via our MCP API.

curl -X GET 'https://glama.ai/api/mcp/v1/servers/sharozdawa/robotstxt-ai'

If you have feedback or need assistance with the MCP directory API, please join our Discord server