Server Quality Checklist

Tool Scores

• validate_tool_argsA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite no annotations, the description transparently states the return structure ({valid, error?, retry_hint?}) and explains that retry_hint is a ready-to-send LLM feedback message. This covers the behavior well, though it omits details like idempotency or side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence with a clear structure: first states the action, then lists the return fields. It is concise but could be improved by front-loading the most critical information (e.g., validation outcome).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the basic function and return values, but lacks usage context like when to validate or how the shape spec works (already in schema). No output schema means description should be more detailed, but it adequately explains the return format.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with detailed descriptions for each parameter. The description adds minimal extra meaning beyond the schema (e.g., clarifying retry_hint's purpose), meeting the baseline for high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Validate') and resource ('tool-call args object against a small shape spec'), clearly distinguishing it from sibling tools (generate_retry_message, lint_tool_definition) which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives. The description implies usage for validating tool-call arguments but does not provide prerequisites or contrasting scenarios with siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_retry_messageA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It describes inputs and output but does not disclose behavioral traits like idempotency, side effects, or read-only nature. The tool appears to be a pure function, but this is not explicitly stated. Minimal transparency beyond functional description.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences that are concise and front-loaded. Every sentence provides essential information without redundancy. The structure is clear and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple string generation tool with 3 parameters and no output schema, the description covers inputs and the output's purpose. However, it does not describe the output format or provide examples, which would be helpful for an agent. Given the tool's simplicity and lack of annotations, the description could be more complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% (no parameter descriptions in schema). The description names the three parameters (tool_name, validation_error, attempted_args) and indicates their roles, but does not elaborate on expected formats, constraints, or examples. It adds value over the raw schema but does not fully compensate for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly specifies the tool's purpose: building a canonical LLM-facing retry feedback message given tool_name, validation_error, and attempted_args. It includes the formatting method (agentvet's ToolArgError.toLLMFeedback()), and distinguishes from siblings (lint_tool_definition, validate_tool_args) by its unique output.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use this tool—when generating a retry message after a validation error. It notes that the wording matches runtime callers, but does not explicitly state when not to use it or compare to alternatives. However, the siblings are sufficiently different, so no confusion.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lint_tool_definitionA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It describes the analysis but does not explicitly state it is non-destructive or disclose output format. The behavior is implied but not fully transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence with no unnecessary words. It efficiently communicates the tool's purpose and checks.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema is provided, and the description does not explain the return value (e.g., a list of issues). For a linting tool, this is a gap that reduces completeness for an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and the parameter 'tool' has a description explaining its structure. The description adds context about what to include, aiding correct invocation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly specifies the verb 'Sanity-check' and the resource 'tool definition', listing specific checks. It distinguishes from sibling tools (generate_retry_message, validate_tool_args) by its focus on definition quality.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for checking tool definitions, but lacks explicit when-to-use or when-not-to-use guidance. However, the context of sibling tools helps differentiate usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.