Server Quality Checklist

Tool Scores

• quality_gateA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses behavior: checks failures for single file, adds flaky check for multiple files, clarifies thresholds are caller-defined, and notes final decision is human. No destructive or side-effect info needed as it's a read-only analysis tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured into purpose, usage, args, and returns. No unnecessary sentences. However, the first sentence could be slightly tighter (e.g., 'Decides pass/fail on JUnit reports with explicit reasons'). Still, it earns its space.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (4 params, output schema exists), the description covers usage scenarios, parameter meanings, and return fields. Minor gap: does not explain exact flaky rate calculation, but this is acceptable as an implementation detail. Overall sufficient for an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 0% description coverage, so the description must compensate. It provides thorough explanations for all 4 parameters: path (with glob example), max_failures, max_flaky_rate (with range and applicability), min_runs (with default). This adds substantial meaning beyond the schema's basic types and defaults.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool decides pass/fail on JUnit reports, differentiating it from sibling tools that provide raw data (flakiness_report, parse_junit). It specifies the resource (JUnit reports) and the action (decide pass/fail with explicit reasons).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use (actionable decision vs raw data) and describes behavior with single vs multiple runs. Lacks explicit 'when not to use' but provides sufficient context through contrast with 'datos crudos' and sibling tool names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• flakiness_reportA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It explains the classification logic, the min_runs threshold, and the return structure. It does not mention any destructive actions, authentication needs, or rate limits, but for a read-only analysis tool, the provided information is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a summary line, usage context, and clearly separated args/returns. It is concise and contains no unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the existence of an output schema, the description provides a sufficient summary of return values and includes all necessary input details. The classification logic and ordering are explained, making it complete for an agent to use correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, but the description adds meaning for both parameters: path_glob is explained with an example and note on alphabetical order, min_runs is explained with default value. This fully compensates for the missing schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: classifies tests as stable, flaky, or broken from multiple successive JUnit reports. It specifies the verb 'clasifica' and resource 'tests', and implicitly distinguishes from sibling tool 'parse_junit' which handles single reports.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says when to use: when you have several successive JUnit reports from the same suite. It provides context about the classification output and mentions 'datos_insuficientes' for cases with insufficient runs, but does not explicitly state when not to use or directly name alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• parse_junitA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavioral traits. It does so by stating that if the file doesn't exist or XML is malformed, the tool fails with an explanatory message. It also outlines the return structure. However, it doesn't explicitly state that the tool is read-only (non-destructive), which is a minor gap but acceptable given the context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is moderately concise, with a clear first sentence, usage guidance, and parameter details. It could be slightly more compact (e.g., merging the parameter and return details), but it remains well-structured and informative without unnecessary fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (1 parameter, output schema exists), the description is complete. It covers purpose, usage, parameter semantics, return structure, and error behavior. The sibling tool is mentioned, providing full context for an agent to understand the tool's role.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema only defines 'path' as a string. The description adds critical semantics: it specifies the path can be absolute or relative to the server directory, and that it must point to a single JUnit XML file. Since schema description coverage is 0%, the description fully compensates and adds value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Parsea UN reporte JUnit XML y devuelve un resumen estructurado.' It uses a specific verb ('parse') and resource ('JUnit XML report'), and distinguishes from sibling tool flakiness_report by noting it is for a single file, while flakiness_report is for comparing runs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says when to use this tool ('cuando tengas la ruta a un archivo de resultados JUnit XML... y quieras saber qué pasó en esa corrida') and provides an alternative for different needs ('Para comparar varias corridas usá flakiness_report'). This clear guidance helps the agent choose correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.