Server Quality Checklist

Tool Scores

• verifyB
  3.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, covering the safety profile. The description adds the behavioral detail that the tool categorizes evidence and proposes an experiment, which is beyond annotations. However, it does not disclose what happens if no evidence is supplied, or whether the proposed experiment is actionable output or just advice. With annotations handling safety, a 3 is appropriate – the description adds some process context but not deep behavioral nuance.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose. No wasted words, and it conveys the two main phases (categorization and proposal). It is slightly abstract but structurally sound. Loses a point for not being more concrete about the tool's operation.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 6 parameters, a nested evidence array, and no output schema, the description is too high-level. It does not explain how an agent should structure input, what kind of 'decisive experiment' is produced, or what the return value looks like. The schema covers parameters, but the overall workflow and expected output remain ambiguous. An agent would struggle to know what to pass beyond the required claim and how to interpret the result.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with each parameter having its own description (e.g., 'claim' described as 'The technical claim to examine'). The tool description adds no parameter-specific details, leaving the schema to carry all parameter semantics. Baseline 3 is correct since the schema fully documents inputs and the description does not contradict or supplement it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action: separating claim-related information into categories and proposing a decisive experiment. The verb 'Separate' combined with 'propose' gives a precise enough purpose. It does not explicitly contrast with sibling tools like ackrite or reality_check, but the title 'Verify a claim' reinforces the intent. Scores 4 rather than 5 because the description could be more explicit about the tool's primary function being analysis of a claim rather than external verification.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus its siblings (ackrite, audit, prove_it, reality_check). It does not mention prerequisites, alternative tools, or scenarios where verification should be attempted via other means. The only implicit usage context is the word 'verify', but that is weak and does not help an agent choose among similar tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ackriteA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already signal read-only, idempotent, non-destructive behavior, so the description does not need to repeat safety traits. It adds value by promising concrete outputs—support, contradiction, assumptions, missing evidence, and next action—which is useful context beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is one front-loaded sentence with no filler; the primary action appears first and the output summary follows. Every clause earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given six parameters and no output schema, the description gives a helpful list of result categories but not their structure or semantics. It leaves the agent to infer how evidence, context, and output fields interrelate, so it is minimally viable but not complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline applies; the description does not introduce meaning beyond what the schema already provides. It merely names claim and evidence, which are also documented in the input schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific action ('Classify... from supplied evidence') and names the object (technical claim), so the purpose is immediately clear. However, it does not contrast itself with the sibling tools verify, audit, prove_it, and reality_check, so it lacks explicit differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  'From supplied evidence' implies the tool is appropriate when evidence is available and a claim needs classification, but there is no when-not-to-use guidance or mention of alternatives. The boundary against verify/audit/prove_it/reality_check is left entirely to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• auditA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, and the description does not contradict these. It adds the audit dimensions, which clarifies what the tool evaluates, but it does not disclose broader behavioral traits such as output format, failure modes, or assumptions beyond the listed criteria.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single dense sentence that front-loads the action and packs the six audit dimensions into a compact list. Every phrase contributes meaningful guidance and nothing is redundant.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is sufficient to understand the tool's high-level purpose but omits anything about expected output, return value, or how evidence and proposedChanges should be used. Given the tool's complexity and lack of an output schema, some additional context would help an agent call it effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description does not add parameter-specific guidance, which is acceptable at the baseline of 3 because no parameter meaning is left undocumented.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description begins with the specific verb 'Challenge' and the resource 'a proposed code change,' then enumerates the exact audit dimensions (scope creep, rewrites, contract assumptions, security, error handling, validation). This makes the tool's purpose unmistakable and differentiates it from a plain 'verify' or 'prove' tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly implies this tool is for critically reviewing proposed implementations, but it does not state when to prefer it over siblings like verify, prove_it, or reality_check, nor does it mention when not to use it. Usage context is present but alternatives are not addressed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• prove_itA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnly, idempotent, non-destructive, and closed-world behavior. The description still adds value by stating the output is deliberately minimal ('smallest...direct') and excludes speculation, which clarifies that the tool will not invent evidence or padded plans. No contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, grammatically tight, with the key constraints front-loaded and no filler. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with seven parameters and no output schema, the description captures the high-level purpose but leaves out what the returned proof plan actually contains and how optional evidence/context influence the output. The title and schema help, but the description alone is not fully sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All parameters have schema descriptions (100% coverage), so the baseline applies. The description itself adds no parameter-level details; it only states the overall goal, which the schema already encodes.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Define') and a concrete object ('smallest direct evidence set and experiment') tied to 'technical claim,' so an agent can identify the core action. It does not explicitly distinguish prove_it from the sibling tools, so it stops short of 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'required to establish a technical claim' implies the tool is for planning proof rather than executing verification, but it gives no explicit when-to-use/when-not-to-use guidance and names no alternative sibling such as verify, audit, or reality_check. This is implied usage at best.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• reality_checkA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is covered. The description adds value beyond these by naming the categories of failure the tool inspects, giving the agent a concrete sense of what the tool evaluates. It does not contradict the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single dense sentence front-loads the tool's purpose and enumerates its detection targets without wasted words. It is appropriately sized for a tool whose parameters are already documented in the schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is adequate for understanding what the tool inspects, and the schema covers all parameters clearly. However, with no output schema, the description does not explicitly state what the tool returns (e.g., a list of findings, a pass/fail result), which is a notable gap for an agent deciding how to use the result.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with each parameter already documented meaningfully (attempts, evidence, reasoning, sessionId). The description adds minimal parameter-specific detail, but because the schema carries the burden, no further compensation is needed. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb ('Detect') and enumerates the exact pathologies it looks for: repeated failed strategies, contradictions, unsupported conclusions, untested success claims, and premature rewrites. This clearly states the resource being analyzed (reasoning and session history). It does not explicitly distinguish itself from the sibling analysis tools, but its function is unique enough to be identifiable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool by referencing 'supplied reasoning and session history', which tells the agent what input should accompany the call. However, it offers no explicit guidance on when not to use it or what alternatives like ackrite, verify, audit, or prove_it are better suited for.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.