Server Quality Checklist

Tool Scores

• improve_test_coverage_orchestratorC
  2.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. However, it does not disclose behavioral traits such as side effects (e.g., file modifications), authorization needs, or what constitutes 'prioritized next actions'. The description is too vague to inform the agent about consequences.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, which is concise, but it sacrifices informativeness. It could be improved by adding key details without becoming overly long.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 10 parameters, no output schema, and no annotations, the description is severely incomplete. It omits the iterative nature, convergence criteria, interaction between parameters, and the meaning of 'prioritized next actions'.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 20%, yet the description adds no parameter details. It does not explain the role of any of the 10 parameters (e.g., targetLinesPercent, maxIterations), leaving the agent without guidance on how to populate them.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Drive the full iterative coverage improvement loop and produce prioritized next actions' conveys a high-level process but lacks specificity. It does not clearly distinguish from sibling tools like ensure_coverage_setup or run_tests_with_coverage, which also cover parts of the loop.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this orchestrator versus the individual sibling tools. There is no mention of prerequisites, alternatives, or context where this tool is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verify_coverage_targetC
  2.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the full burden. It says 'compare' but does not disclose whether the tool is read-only, what happens on threshold pass/fail, or any side effects. Minimal behavioral info.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is only one sentence, which is concise but at the cost of missing essential details. It is not front-loaded with critical info like parameter hints or return behavior.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 6 parameters, no output schema, and no annotations, the description is far from complete. It only gives a high-level purpose without addressing return format, side effects, or parameter details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is only 33%, and the description adds no extra information about any of the 6 parameters. It does not explain what the target percentages represent or any constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'compare' and the resource 'current coverage metrics against requested thresholds'. It distinguishes the tool from siblings (setup, fetch, improve, run tests), but could be more specific about the source of current metrics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines1/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. No mention of prerequisites, when to apply, or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fetch_uncovered_codeC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description must fully disclose behavior. It mentions importing and fetching but omits side effects, permissions, or whether the import modifies state. Minimal transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Extremely short, but not overly concise to the point of losing necessary detail. It is front-loaded but lacks substance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 7 parameters, no output schema, and no annotations, the description is severely incomplete. It does not explain what 'ranked uncovered symbols' means or how results are structured.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 29%, and the description adds no parameter explanations. Parameters like limit, language, nodeType are left undefined, forcing the agent to guess.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description specifies the main actions: importing LCOV and fetching ranked uncovered symbols. It is clear enough to understand the tool's role among siblings, though it could be more precise about the output.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like verify_coverage_target or run_tests_with_coverage. The description does not state prerequisites or context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_tests_with_coverageC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description must fully disclose behavior. It only states it 'runs tests and coverage and collects failures plus artifact paths,' omitting whether it modifies files, requires permissions, or has side effects. This is insufficient for safe invocation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence of 15 words, front-loaded with the action. Every phrase earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 4 parameters, no output schema, and no annotations. The description fails to explain parameter interactions, output format, or error handling, making it incomplete for reliable agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 50% schema description coverage, the description should clarify parameter usage. It does not explain testCommand or coverageCommand format, and does not distinguish between workspacePath and workspacePaths. Minimal added value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it runs tests and coverage and collects failures and artifact paths. This distinguishes it from sibling tools like ensure_coverage_setup or fetch_uncovered_code, though it doesn't explicitly name them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like ensure_coverage_setup or verify_coverage_target. The description lacks context about prerequisites or limitations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ensure_coverage_setupC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided. Description only says 'ensure' without disclosing side effects (e.g., installing dependencies, modifying workspace). Bar is high without annotations; this falls short.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, front-loaded with purpose. No unnecessary words, but could expand slightly without losing conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 4 parameters, no output schema, and no annotations, the description is too brief. Does not explain what 'coverage commands' or 'dependencies' means, nor success/failure criteria. Incomplete for a setup tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 50% with descriptions for two parameters, but the description adds no parameter-level meaning. The optional parameters preferredRunner and preferredCoverageProvider are unexplained beyond their names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it ensures the target workspace has runnable coverage commands and dependencies. The verb 'ensure' plus specific resource and outcome distinguish it from sibling tools like fetch_uncovered_code (data retrieval) or improve_test_coverage_orchestrator (improvement).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this vs alternatives. Missing context about prerequisites, idempotency, or when setup is needed. Sibling tools exist but no differentiation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.