Server Quality Checklist

Tool Scores

• find_leakB
  3.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It does disclose the analytical behavior: identifying secret inputs that contribute to the completion signal and reporting the fan-in cone size. However, it does not mention side effects, required inputs, error handling, or output format, leaving gaps for an agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no filler. It efficiently communicates the core action and expected result, making it easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 4 parameters, no annotations, and no output schema, yet the description provides only the core purpose. It omits parameter details, usage guidance, and output structure, making it insufficient for an agent to invoke the tool correctly with confidence.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not explain any of the four parameters (verilog, secrets, module, observation) or their roles. 'Secret inputs' vaguely relates to `secrets`, and 'completion signal' might relate to `observation`, but the description adds no usable parameter-level semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Localise') and resource ('timing leak'), and clearly states the tool's output: naming secret inputs that reach the completion signal and the fan-in cone size. This strongly distinguishes it from sibling tools like check_constant_time, which likely verifies rather than localizes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives no explicit guidance on when to use this tool versus alternatives such as check_constant_time or check_masking. The phrasing 'Localise a timing leak' implies a use case, but no context, exclusions, or alternative tool references are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_masking_gadgetsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It implies a read-only operation but does not disclose return format, side effects, or limitations. Minimal behavioral context is added beyond the tautological 'list'.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is one concise sentence, front-loaded with 'List'. However, the phrase 'for your own' is ambiguous and could be clearer, slightly reducing structure quality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with no parameters, annotations, or output schema. The description states what is listed (masking gadgets and JSON netlist format) but does not describe the return structure or any behavioral details. Adequate but minimal for a list tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline for parameter semantics is 4. The description does not need to explain any parameters, and no parameter details are missing.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'List' and the resource 'bundled masking gadgets', and also mentions 'JSON netlist format', giving a specific scope. It distinguishes from siblings like 'list_benchmark_fixtures' by the resource, though not explicitly.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance is provided on when to use this tool versus alternatives. The listing nature implies use when needing the gadget list, but no context or exclusions are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_reference_checkerB
  3.2/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full responsibility for behavioral disclosure. It merely states that the tool runs a baseline, with no mention of outputs, side effects, duration, or required permissions. This is completely silent on behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence with no filler. It immediately conveys the action and scope, making it highly concise and readable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  There is no output schema or annotations, and the description does not clarify what the tool returns or what happens after execution. Given that the tool operates on a 'whole benchmark corpus,' the description omits critical context about results, logging, or error handling, making it incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has zero parameters, so semantic explanation of parameters is not needed. The baseline of 4 for zero-parameter tools applies; however, no additional context about implicit inputs (like configuration) is given, preventing a higher score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Run') with a clear resource ('bundled cone-of-influence baseline') and scope ('whole benchmark corpus'). This distinctly identifies the tool's action and differentiates it from sibling tools that focus on specific security checks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like check_constant_time or find_leak. The description does not mention exclusions, prerequisites, or scenarios that would favor this baseline run.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• prove_confidentialB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the key behavior of proving without disclosure, but it fails to explain what happens when called, what inputs are needed (though none are in schema), what output to expect, or any side effects. 'Call it to see what is' is cryptic and does not clarify behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the primary purpose. The second sentence about availability is relevant context but not redundant. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters and no output schema, the description is all the context. It lacks details about what kind of property can be proven, who the third party is, or what the tool returns when called. The availability note adds intrigue but not functional completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so baseline is 4. The description does not need to add parameter meaning; the empty schema is self-explanatory.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool proves a property to a third party without disclosing the design, using a specific verb and resource. It distinguishes itself from sibling tools focused on checking constant time, leaking, or masking by emphasizing non-disclosure.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The second sentence about availability in the open-source distribution is not a usage guideline and does not mention sibling tools or exclusion criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_benchmark_fixturesA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It describes the corpus contents but does not state whether the operation is read-only, what the return format will be, or any side effects. The verb 'List' implies a non-mutating operation, but this is not explicit, and no output details are given.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, front-loaded with the action and resource, followed by a concise explanation of the corpus. Every word adds meaning, with no redundant or filler content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with no output schema, the description explains what the corpus is but does not specify what the actual list output contains (e.g., fixture IDs, names, metadata). Given the lack of annotations and output schema, the description is partially incomplete for an agent to fully anticipate the tool's output.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline is 4. The description adds valuable context about the corpus structure, which helps an agent understand what the list will contain, even though no parameter explanations are needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'List' and identifies the exact resource: the ctbench matched-pair corpus. It also explains the corpus's composition (constant-time designs paired with leaky twins and a control), which clearly distinguishes it from sibling tools like get_benchmark_fixture that likely retrieve individual fixtures.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for browsing or enumerating the available benchmark fixtures, but it does not explicitly state when to use this tool versus alternatives like get_benchmark_fixture or list_masking_gadgets. There is no mention of exclusions or prerequisites, but the context of 'listing' is clear enough to infer basic usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• replay_certificateA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It discloses a key behavioral trait (integer arithmetic only, no solver) that goes beyond the schema. However, it omits other behavioral aspects like output format, side effects, or error handling, leaving some gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose and followed by a concise rationale. No filler or redundant content is present.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Even though the tool is simple, the description lacks essential context about what the tool returns (no output schema) and how it handles invalid certificates. Without this, an agent cannot know the expected result or verify success, making the description incomplete for invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It does identify the certificate as an 'elimination certificate', giving it semantic meaning, but does not explain the structure or required fields of the object. This is minimal compensation for a single unstructured object parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action 'Re-check' and the resource 'elimination certificate', and specifies the method 'integer arithmetic only'. This distinguishes it from sibling tools like 'check_constant_time' or 'find_leak', which target different verification aspects.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for verifying someone else's claim without trusting them or an SMT solver, giving clear context. However, it does not explicitly name alternative tools or state when not to use this one, so it falls short of full exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_benchmark_fixtureA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must carry the behavioral disclosure burden. It indicates a read-only fetch operation and that fixtures are bundled locally, but it does not address potential errors (e.g., non-existent name) or the exact return format. This is basic transparency for a simple read tool, but lacks depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that immediately states the action and object, with no filler or redundancy. Perfectly concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one parameter and no output schema, the description sufficiently defines the purpose and input. It could mention error handling, but the simplicity of the tool makes the description adequate for an agent to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers the single parameter 'name' with a 100% description and an example format. The tool description adds minimal extra meaning by linking the parameter to 'bundled benchmark fixture', but overall the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Fetch' and clearly identifies the resource: the Verilog source of a bundled benchmark fixture. This distinguishes it from siblings like list_benchmark_fixtures, which lists fixtures rather than fetching their content.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for retrieving the source code of a named fixture, but it does not explicitly mention when to use this versus alternatives such as list_benchmark_fixtures. No exclusions or alternative guidance is provided, making it merely implied usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• score_benchmark_submissionA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must carry the full burden. It discloses the key behavioral trait that it reports two categories of errors separately and the reason, which adds value. However, it does not disclose output format, side effects, or how unknown verdicts are treated, leaving gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, immediately stating the action and then providing a concise rationale for the report separation. No filler words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with one well-described parameter, and the description explains the main output distinction. However, without an output schema, it would be helpful to specify the return format or how UNKNOWN verdicts are handled, leaving some incompleteness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides full coverage of the single parameter, describing it as a mapping from fixture file name to verdict values. The tool description adds no extra parameter details beyond 'set of verdicts', so the baseline score applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool grades a set of verdicts against the benchmark, using the verb 'Grade' and specifying the resource. It distinguishes itself from siblings by focusing on evaluation of verdicts and explicitly separates unsound from imprecise verdicts, which no other sibling suggests.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides some context by explaining why unsound verdicts are reported separately ('because only the first kind ships a vulnerability'), but it lacks explicit guidance on when to use this tool versus alternatives like check_constant_time or find_leak. The usage is largely implied from the name and description.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_constant_timeA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description covers key behavioral traits: it returns a verdict, identifies leaking secrets, and cannot be overridden by the user. It stops short of describing all side-effect or edge-case behavior, but for a read-only checker this is a strong disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two compact sentences convey purpose, output, and a behavioral constraint without redundancy or filler. The most important information is front-loaded in the first sentence.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Without an output schema, the description correctly includes return semantics. It covers the core workflow (input Verilog and secrets, get a verdict) and the crucial re-run behavior; optional parameters are already documented in the schema, so the description is sufficiently complete for the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds only light semantic reinforcement by referring to 'declared secret inputs' and a 'completion signal', which align with the secrets and observation parameters but do not introduce new syntax or format details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Decide') and resource ('Verilog module's completion signal') and clearly distinguishes the tool's goal from siblings like check_masking or prove_confidential. It also states the output values (CONSTANT_TIME/LEAKY), making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It clearly implies the use case: determining whether completion depends on declared secrets. It also adds a practical guideline ('re-run this tool after any fix') and a constraint ('cannot declare constant-time yourself'), though it does not explicitly compare with alternative sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_maskingA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavioral traits. It explains the two certificate mechanisms (dependence and uniformity) and the failure output (naming the probe wire that recombines a secret). This goes beyond a simple 'checks security' statement and gives the agent a clear model of what the tool does internally.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, information-dense, and front-loaded with the core purpose. Every clause adds value: input types, certificate types, and failure behavior. No wasted words or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is complete for a tool with one parameter and no output schema. It explains inputs, the verification procedure, and the failure output. It does not detail the netlist object structure, but the schema's 'netlist spec object' covers that. Slight gap in not specifying the success output format, but the certificate names imply it. Overall adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for the single parameter, and the description echoes the same info ('bundled gadget name' vs 'JSON netlist'). The description adds only marginal clarity by calling it a 'JSON netlist' instead of 'netlist spec object'. Since the schema already provides full semantics, a baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to certify a masked gadget as first-order secure under glitch-free probing, or to identify the probe wire that recombines a secret. This is a specific verb+resource+outcome, and it distinguishes itself from siblings like check_constant_time (timing) and find_leak (leakage finding).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context: it accepts a bundled gadget name or a JSON netlist, and it explains the two certificate types tried. It does not explicitly mention when not to use it versus alternatives, but the specialized nature makes the intended use obvious. Missing explicit exclusions or alternative pointers, so not a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_patch_completeA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full responsibility for behavior. It discloses the three possible return values (COMPLETE, INCOMPLETE, VACUOUS), explains what INCOMPLETE includes (a surviving violating input), and clarifies the meaning of VACUOUS. This is excellent transparency for a decision-making tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the main purpose and immediately providing key behavioral outcomes. Every sentence is informative, with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description fully covers the tool's purpose and outputs, which is critical since there is no output schema. It explains the three-way result and what INCOMPLETE returns. It lacks explicit prerequisites or a 'when not to use' statement, but the context is sufficiently complete for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for the single parameter defect_class, so the baseline is 3. The description does not add any parameter-specific details, but the schema already provides an example ('e.g. A, B, C'). No additional semantics are needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with a specific verb ('Decide') and resource ('modelled bounds-check repair'), and explicitly distinguishes its scope ('eliminates EVERY violating input, not just a known one') from simpler checks. This differentiates it from sibling tools like check_constant_time and check_masking.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use this tool: when you need to ensure a repair is complete across all violating inputs, rather than testing a single known case. It does not explicitly name alternatives or exclusions, but the context is clear enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_defect_classesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full transparency burden. It discloses the content (defect classes and exclusions) and hints at the tool's non-destructive nature. It doesn't explicitly state read-only or side-effect-free, but for a list operation this is reasonably transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence starting with the verb 'List'. Both clauses earn their place: the first specifies the main output, the second clarifies an important subtlety about verdict coverage. No filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (no parameters, no output schema), the description fully covers the essential context. It explains what the tool returns and the specific caveat about COMPLETE verdict, which is sufficient for an agent to understand its purpose and output.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline of 4 applies. The description adds no parameter-specific information, but none is needed since there are no parameters to document.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('list') and resource ('modelled bounds-check defect classes'), and adds a clarifying nuance about what a COMPLETE verdict does not cover. This clearly distinguishes it from sibling tools that perform checks or calculations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The context is clear: use this when you need an overview of defect classes or to understand the scope of a COMPLETE verdict. It doesn't explicitly name alternatives or exclusions, but the listing nature and unique focus differentiate it sufficiently.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.