Server Quality Checklist

Tool Scores

• run_checksB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the behavioral disclosure burden. It states execution is 'in the sandbox' (an environmental detail) and references the verdict shape of 'verify', but it does not disclose whether the tool is read-only, whether it modifies anything, what side effects exist (if any), or any authorization requirements. For a tool with no annotation coverage, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, front-loaded with the key action, and contains zero filler. Every element ('run lint/type checks', specific tools, sandbox, shape reference) earns its place. Excellent conciseness and structure.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  While the tool is simple (one optional param, output schema present), the description leaves critical gaps: it does not explain the 'checks' parameter, relies on the reader knowing 'verify's verdict shape' (which may be defined elsewhere), and does not clarify whether all checks run or a subset can be requested. An agent may not be able to call the tool correctly without additional inference.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter, 'checks', is entirely undocumented in the schema (0% coverage). The description mentions the tools 'ruff' and 'mypy' but never explains that 'checks' likely selects or filters which of these to run, nor does it describe the expected values. The description adds no meaning beyond the schema, leaving the parameter's role ambiguous.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('run lint/type checks') and specifies the exact tools involved ('ruff', 'mypy'), and notes the 'same verdict shape as verify' to differentiate from at least one sibling. It gives a specific, unambiguous purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies a context (running lint/type checks) and points to a related tool ('verify') for shape, but it does not explicitly state when to use this tool over 'verify' or the other siblings. There is no direct 'when not to use' guidance, though the intent is somewhat inferable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verifyA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It reveals that tests run 'in a sandbox', implying isolation, which is useful. However, it does not mention whether the tool modifies files, requires network access, or has any side effects on the workspace. Since it does not contradict annotations (none exist) and provides one meaningful behavioral detail, a middle score is appropriate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, front-loaded sentence that immediately communicates the action, target, and method. Every word contributes value, with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is relatively simple: two optional params, no required fields, and an output schema exists (so return format need not be explained). The description covers the core behavior adequately. While it could elaborate on how 'base' and 'scope' interplay, the schema already documents those, so the description does not need to repeat them. Minor gap: it does not state whether the tool is read-only, but the sandbox hint mitigates that.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% – both 'base' and 'scope' have descriptions in the schema. The tool description adds no additional parameter semantics beyond what the schema already provides. Per the baseline rule for high schema coverage, a score of 3 is given.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Verify'), the target ('current working-tree changes'), and the method ('running affected pytest tests in a sandbox'). It distinguishes itself from siblings: explain_failure is for diagnosing failures, history is for past actions, and run_checks is a broader term that might overlap, but this tool is explicitly about pytest verification, making its niche unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for verifying changes before committing, but it does not explicitly state when to prefer this over run_checks or provide exclusion criteria. There is no mention of alternatives or conditions that would make this tool inappropriate, leaving the routing decision to the agent's inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explain_failureA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full disclosure burden. It reveals the return content (traceback and output) and the source (previous verify run), which is helpful. Yet it does not explicitly declare read-only behavior or lack of side effects, nor what happens if the check_id is not found. This is a moderate gap given the absence of annotation support.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficiently worded sentence that front-loads the core purpose and key constraint (from a previous verify run). There is zero wasted text, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema is present and parameters are fully described, the description provides sufficient context for the tool's purpose and usage. It mentions the prerequisite of a prior verify run and the focus on a specific check. Minor omissions like error handling or explicit read-only status are not critical for basic invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with both run_id and check_id already documented, including the default behavior of run_id and an example for check_id. The description adds no additional semantic value beyond the schema, such as clarifying the format of output or the relationship between the parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description precisely states the tool's function: retrieving the full traceback and output for a single failing check from a prior verify run. This clearly distinguishes it from siblings like 'verify' (which runs checks) and 'history' (which lists runs), as it focuses on failure details.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides context by specifying the tool is for a 'previous verify run', implying it should be used after a verification execution. However, it does not explicitly name alternatives or state when not to use it, leaving some room for inference about choosing it over history or re-running verify.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• historyA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It reveals that the tool returns historical information (first/last seen, count) and implies a read-only query, but it does not explicitly state read-only status, error handling, or any side effects. For a simple query tool, this is acceptable but not exhaustive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with zero filler. The first sentence directly states the query, and the second explains the use case. It is front-loaded with the core purpose and efficiently conveys essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one required parameter, output schema exists) and the description's coverage of purpose and usage, it is largely complete. It could explicitly mention that the tool is read-only, but that omission is minor given the lack of annotations and the straightforward nature of the operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema defines 'fingerprint' as a plain string with no description (0% coverage). The tool description adds meaning by referring to it as a 'failure fingerprint' and using it in context, clarifying that it identifies a specific failure pattern. This compensates for the lack of schema documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states what the tool returns (first/last seen, count) and identifies the resource (failure fingerprint). It also gives a specific use case—distinguishing new regressions from long-standing breakage—which clearly differentiates it from siblings like verify or explain_failure.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a clear when-to-use: when you need to know if a fingerprint is unknown (possible regression) or has been seen across runs/commits (long-standing breakage). It gives a concrete scenario but does not explicitly name alternatives or state when not to use the tool, so it falls just short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.