Server Quality Checklist

Tool Scores

• estimate_auditA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full transparency burden. It reveals that no full mutation cycle runs, that Rust counts are exact cargo-mutants candidates, that other languages use a heuristic labeled 'approx', and that withTiming:true adds a test-suite run. It does not mention errors, permissions, or edge-cases, but the main behavioral trade-offs are well disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense and front-loaded: purpose first, then key behavioral caveats, then usage guidance. The parenthetical about Rust versus TS/JS/Python/PHP is technical but earns its place since that fidelity distinction is essential for acting on the estimate.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists and all parameters are documented in the schema, the description covers purpose, behavior, language-specific fidelity, and integration with a sibling tool. It could add preconditions or failure modes, but it is substantially complete for this tool's scope.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and the schema already thoroughly documents filePath, timeoutMs, and withTiming. The description adds some context around withTiming and the estimate's relationship to audit_code_resilience, but it does not meaningfully expand on the schema's parameter details. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: it is a 'Cheap pre-flight estimate' of auditing a file, and it explicitly distinguishes itself by noting it runs WITHOUT the full mutation test cycle. It also differentiates from sibling audit_code_resilience by framing this tool as the pre-flight decision aid before running that audit.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly says 'Use this before audit_code_resilience to decide whether to audit now, scope down, or skip,' giving explicit usage context and naming the primary alternative. It lacks an explicit 'when not to use' or comparison with triage_test_coverage, so it falls just short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• audit_code_resilienceA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description takes full responsibility for behavioral disclosure. It explains the sandbox isolation, on-demand execution, generation of logical mutants, and the nuanced path resolution behavior (`filePath` vs `target`/`workspace`). This goes beyond a simple statement of purpose and gives the agent operational expectations about environment and output paths.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is moderately long but each sentence carries useful information: purpose, mutation mechanics, language support, and the critical path-resolution caveat. It is front-loaded with the main purpose. While the path section is dense, it addresses a real gotcha and is not filler. It is slightly longer than strictly necessary but never wasteful.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex tool with 21 parameters and an output schema, the description supplies the essential context: what the tool does, how it behaves in a sandbox, supported languages, and the subtle path semantics. It also references a sibling tool's output as a valid input, connecting the workflow. The output schema and parameter descriptions cover the remaining details, so the description is complete without needing to restate schema content.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides 100% coverage of all 21 parameters with detailed descriptions, so the baseline is 3. The tool description adds crucial semantics for `filePath`: it is resolved against the server's working directory, and the result's `target` is relative to the audited file's own workspace, with guidance on reconstructing a usable `filePath`. This extra layer of context for a key parameter justifies a score above baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Runs on-demand, sandbox-isolated mutation testing against a single source file to identify gaps in unit test coverage.' It further explains the mutation generation and test-suite interaction, making the tool's function unambiguous. It also differentiates from siblings by focusing on coverage-hole detection via mutation testing and even references triage_test_coverage's output as valid input, showing awareness of the tool ecosystem.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It clearly states when to use the tool: to audit a single file for test coverage gaps via mutation testing, with per-language support. It implicitly positions itself as a follow-up to triage_test_coverage by noting that a `file` from that tool is a valid `filePath`. However, it does not explicitly mention when not to use it or name alternatives like estimate_audit, so it falls short of a full 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• triage_test_coverageA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses key behaviors: directories are recursively expanded, test files are skipped, files are audited in parallel with a shared wall-clock budget, and the output format includes 'unaudited' files when the budget runs out. It also warns about resource intensity ('a sweep is still the most resource-hungry thing this server does') and the Rust exception. The only minor gap is that it doesn't explicitly state whether the tool is read-only or has side effects, but the description implies it's a read-only audit operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the core purpose. It uses paragraphs to separate the main description, parameter details, and cross-tool guidance. It is somewhat long but each sentence adds value, covering key behaviors, parameter interactions, and usage guidance. The only minor issue is that it could be slightly more concise, but the density of information justifies the length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (10 parameters, no required params, output schema present), the description is remarkably complete. It covers the main use case, parameter interactions, edge cases (Rust memory, timeouts), and cross-tool integration. The output schema exists, so the description doesn't need to explain return values in detail, but it does mention the 'unaudited' field and the gate behavior. The description is sufficient for an agent to use the tool effectively without additional context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters. The description adds value by explaining the interaction between parameters (e.g., diffBase + paths intersection, maxFiles precedence, totalTimeoutMs clamping per-file timeout, fileConcurrency effects on different engines). It also provides examples and clarifies the meaning of 'file' relative to the working directory, which is crucial for passing to audit_code_resilience. The description goes beyond the schema by explaining the resource implications of fileConcurrency and the behavior of totalTimeoutMs.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Batch triage: audit a set of files and/or directories and return a weakest-first ranked leaderboard of mutation scores'. It specifies the verb (audit/triage), the resource (files/directories), and the output (ranked leaderboard). It also distinguishes from siblings by mentioning 'Drill into a weak file with audit_code_resilience' and noting the difference in parameter spelling.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidance: it explains when to use this tool (for a batch overview of fragility) and when to use the alternative (audit_code_resilience for per-mutant detail). It also explains the diffBase option for auto-scoping to changed files, and the precedence of maxFiles. The description clearly states the relationship with sibling tools, making it easy for an agent to choose correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.