robots.txt Check

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare read-only, idempotent, and non-destructive. The description adds substantial behavioral context: RFC 9309 compliance, handling of unparseable lines, and Google's interpretation of 5xx as 'disallow everything.' It also clarifies that only the origin of a URL is used. No contradictions with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is tightly structured with clear sections: overview, args, returns, example, and alternative tool. Every sentence provides value, including the RFC reference, the deindexing flags, and the return structure. No filler or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a read-only analysis tool, this description covers behavior, parameters, return fields, an example, and a pointer to an overlapping sibling. It also includes important SEO context (deindexing risks) and complies with RFC 9309. The Returns line enumerates top-level fields, so agents understand the output shape without a formal output schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100% — both `site` and `response_format` are fully described in the input schema, including examples and defaults. The description restates these parameters and adds an example, but does not materially expand beyond the schema's own documentation, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource: 'Fetch and parse a site's robots.txt per RFC 9309.' It enumerates exact outputs (user-agent groups, Allow/Disallow rules, sitemaps, unparseable lines) and flags two critical failure modes. It also distinguishes itself from sibling `ai_crawler_access` by explicitly directing AI-crawler queries to that alternative.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides a concrete example ('What does example.com's robots.txt allow?') and an explicit alternative tool for AI-crawler specifics, establishing clear when-to-use and when-not-to-use boundaries. The mention of deindexing flags also signals relevant SEO audit scenarios, giving strong contextual guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.