fetch_robots

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, and non-destructive. The description goes beyond by detailing the matching rule ('Google-style longest-match with Allow-wins-on-tie') and listing what it returns (raw robots.txt, parsed groups, sitemap refs, verdict). This adds useful behavioral context beyond the annotation tags, with no contradiction.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

Two concise sentences: the first states the core action and output, the second details the matching rule and return payload. No filler, and the most important information is front-loaded. Excellent structure for an agent to quickly grasp.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 5 parameters and no output schema, the description is reasonably complete: it names the input (origin URL, user-agent), explains the algorithm, and enumerates the return components. The optional parameters are not covered, but their roles are inferable from names (timeout, redirects, private hosts). The core functionality is fully specified, and the safety profile is covered by annotations.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is only 40% (url and user_agent have descriptions). The description clarifies that 'url' is derived origin+path automatically and ties 'user_agent' to matching groups. However, it does not explain timeout_ms, max_redirects, or allow_private_hosts, leaving those unexplained despite low schema coverage. Partial compensation, not complete.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('Fetch and parse') a clear resource ('robots.txt') and an outcome (determine whether a target URL is crawlable). It distinguishes itself from siblings like fetch_sitemap or http_get by focusing on robots.txt semantics, making the tool's role unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description clearly implies its use case — checking crawlability via robots.txt — but it does not explicitly contrast with alternatives like http_get or fetch_sitemap, nor does it state conditions when this tool should not be used (e.g., if only sitemap data is needed). The guidance is implicit rather than explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.