Server Quality Checklist

Tool Scores

• build_release_evidence_packA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and idempotentHint=true, covering safety and repeatability. The description adds that it combines inputs into an artifact but does not explain the return format or whether any side effects occur beyond reading. Since annotations carry the safety profile, a score of 3 reflects the minimal additional behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that front-loads the action and outcome, with no redundant phrases.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The task is moderately complex (combining two data sources), but the description gives only a high-level outcome without specifying the return structure. Since there is no output schema and no mention of output format, the description is adequate but not comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All three parameters (baseRef, reportFiles, repositoryRoot) have schema descriptions with 100% coverage. The description's reference to 'Git changes' and 'JUnit reports' loosely maps to parameters but adds no syntax or format details beyond the schema, so the baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Combine') and identifies both source resources ('Git changes' and 'JUnit reports') and the output components ('risk decision, findings, and rollback checklist'), making the purpose clear. It also distinguishes from siblings by describing a composite operation rather than single-source inspection.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is appropriate when both Git changes and JUnit reports are needed together, which is clear context. However, it does not explicitly mention alternatives or when-not-to-use cases, so it lacks explicit exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• inspect_release_changesA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false. The description adds 'without changing the repository', which reinforces but does not substantially extend the annotations. It does not disclose additional behavior beyond the diff inspection and risk classification, so the added informational value over the structured annotations is limited.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that efficiently conveys the action, scope, and side-effect profile. Every word earns its place; there is no fluff or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only inspection tool with two well-documented parameters and rich annotations, the description adequately covers the core behavior and purpose. It does not describe the return format, but the phrase 'classify release risk' provides a clear expectation, and the absence of an output schema lowers the burden on the description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage for both parameters ('repositoryRoot' and 'baseRef'), each with meaningful descriptions. The tool description adds no parameter-specific details, so it does not go beyond what the schema already provides. The baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Inspect') and names a concrete resource ('local Git diff'), while also stating the higher-level purpose ('classify release risk'). This clearly differentiates it from sibling tools like summarize_test_reports and build_release_evidence_pack, which focus on different operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description establishes a clear usage context: inspecting a local Git diff to assess release risk, with the added safety note that it makes no changes. It does not explicitly name alternative tools or exclusion conditions, but the scope is specific enough for an agent to choose it over the siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• summarize_test_reportsA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnly, idempotent, and non-destructive. The description adds value by explicitly noting that the tool will not return testcase logs or captured output, which is a behavioral trait not covered by annotations. It also scopes the tool to 'local' reports, providing additional context about its operational boundary.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that front-loads the core action ('Summarize local JUnit XML reports') and includes one clarifying exclusion. Every word earns its place; there is no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read-only tool with 2 well-documented parameters and no output schema, the description covers the essential scope and constraints. It does not detail the summary format, but that is not critical given the tool's straightforward nature. It is slightly less complete than ideal because it does not reference alternatives or provide any edge-case guidance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with both repositoryRoot and reportFiles already fully described in the schema. The tool description adds no new parameter meaning beyond what the schema provides, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Summarize local JUnit XML reports' – a specific verb, resource, and scope. It also distinguishes from siblings by specifying it does not return testcase logs or captured output, which is not present in sibling tool names (inspect_release_changes, build_release_evidence_pack).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the usage context (local JUnit XML reports) but provides no explicit guidance on when to use this tool versus its siblings, nor any exclusions. It does not mention 'use this instead of build_release_evidence_pack', so usage is implied rather than stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.