generate_robots_txtA ready-to-paste robots.txt block for one stance, with the reasoning and the crawlers it names. Omit the argument to get all eight stances described. Example: stance='block-ai-training'.
| Name | Required | Description | Default |
|---|---|---|---|
| stance | No | One of allow-all, block-ai-training, block-all-ai, block-datasets, allow-ai-search-only, block-seo-tools, block-disputed, maximum-ai-visibility. |
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark the tool read-only, idempotent, and non-destructive. The description adds useful behavioral context: the output is a paste-ready block, includes reasoning and named crawlers, and changes behavior when the optional argument is omitted. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Two tight sentences carry the full purpose, output format, content summary, and optional behavior. The key statement about producing a robots.txt block is front-loaded, and there is no wasted wording.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single optional parameter with no output schema, the description is nearly complete: it names the output kind, the included reasoning/crawler details, and the default behavior when the argument is omitted. It could add explicit error-handling or stance definitions, but these are minor gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema already fully documents the stance parameter and its allowed values. The description adds meaningful semantics beyond the schema by explaining that omitting the argument returns all eight stances, and provides a concrete example, which helps an agent invoke the tool correctly.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool generates a ready-to-paste robots.txt block for a stance, which is a specific verb and resource. It also clarifies optional behavior (omit argument to get all eight stances) and is clearly distinct from sibling lookup/classification tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context: use it to produce a robots.txt snippet for a specified stance, or omit the argument to get all stances. It does not explicitly name alternatives or when-not-to-use, but the sibling tools are sufficiently different that the intended usage is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Add one secure layer between your agents and this server.
Each tool targets a distinct action: classifying user-agent strings, verifying IP addresses, listing, looking up, generating robots.txt, and checking data freshness. The explicit cross-reference between classify_user_agent and is_verified_crawler_ip clarifies rather than blurs their boundaries.
Almost all tool names follow a predictable snake_case verb_noun pattern such as classify_user_agent, list_crawlers, and lookup_crawler. whats_changed deviates slightly from the verb_noun style but remains consistent in lowercase snake_case, so the naming is mostly coherent.
Six tools is a well-scoped set for a crawler index service. Each tool covers a distinct need without overlapping or unnecessary additions.
The tool surface covers the full workflow for understanding and managing crawler data: discovery, identification via UA or IP, detail lookup, robots.txt generation, and freshness monitoring. No obvious dead ends or missing operations for the stated purpose.