Server Quality Checklist

Tool Scores

• verify_cancellationA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations indicate it is not read-only (readOnlyHint=false) and not destructive (destructiveHint=false). The description adds behavioral context by stating it returns a 'classification and a signed receipt' and implies a side effect of creating a verification record, but does not explicitly describe state changes. Overall, it adds useful detail beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that efficiently conveys the core purpose and output. It is concise without being overly terse, but could be slightly improved by front-loading the key action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the 7 parameters and lack of output schema, the description explains the output (classification and signed receipt) but does not cover error cases, prerequisites (e.g., prior cancellation), or how parameters like operation_id and idempotency_key affect behavior. The schema descriptions fill some gaps, but overall completeness is average.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All 7 parameters have schema descriptions (100% coverage), so the baseline is 3. The description does not add new information about the parameters beyond what's in the schema, except for mentioning the default connector (Stripe) which aligns with the 'connector' parameter. No additional semantic value provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to verify that a subscription claimed to be cancelled is actually cancelled in the system of record (default Stripe). It specifies the action ('verify'), the resource ('subscription cancellation'), and the expected output ('classification and a signed receipt'), distinguishing it from sibling tools like get_verification or verify_refund.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions the default system of record (Stripe) but does not explicitly guide when to use this tool over alternatives like verify_refund or verify_ticket. The usage context is implied by the name and the mention of 'cancelled', but no direct guidance on when not to use it or what the prerequisites are.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_verificationA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, so description adds minor context about returning the receipt. No contradictions; description aligns with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, front-loaded with essential information, no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Simple tool with one parameter and read-only semantics; description adequately conveys the purpose and return value despite lacking an output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema covers the single parameter with a clear description. Tool description does not add new information beyond 'by id', so baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The tool name and description clearly state it fetches a past verification by ID, and includes an example use case. This distinguishes it from sibling tools like verify_cancellation or reconcile_verification.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Description provides an example usage ('to re-read the result or the receipt after the original call') that implies when to use, but does not explicitly exclude alternatives or state when not to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• reconcile_verificationA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations (readOnlyHint: false, openWorldHint: true) are present but the description adds context ('live system of record') and mentions return value ('refreshed classification and receipt'). However, it does not discuss side effects, idempotency, or what happens if the verification ID is invalid.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences: first states the action, second provides usage guidance and return value. No wasted words, front-loaded with key information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one required parameter and no output schema, the description covers purpose and use cases adequately. It mentions return value ('classification and receipt') but lacks details on format, which is acceptable given the simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, with the single parameter verification_id already described. The description adds no new information about the parameter beyond what the schema provides, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('re-run a past verification') and resource ('past verification'), with concrete examples ('catch a completion that has since regressed'). It distinguishes from sibling tools that are for initial verifications or read-only access.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit use cases ('use this to catch a completion that has regressed or recovered'), implying when to use and when not to (new verifications via verify_* tools). It does not explicitly mention alternatives, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verify_ticketA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations indicate readOnlyHint=false and destructiveHint=false. The description adds context about supported systems, return of a signed receipt, and sandbox mode via the test parameter, which goes beyond the annotations. No contradiction exists.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise—two sentences that front-load the core purpose and add key behavioral details. Every sentence adds value with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 8 parameters and no output schema, the description gives a high-level overview and mentions return values (classification and signed receipt), but lacks detailed output format. Still, it provides sufficient context for understanding the tool's role.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so each parameter has a clear description in the schema. The tool description does not add additional semantics beyond what the schema provides, resulting in a baseline score of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'verify', the resource 'support ticket resolution', and specifies the supported systems of record. It distinguishes from siblings like verify_cancellation and verify_refund by focusing on resolution state.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when an agent claims resolution, but it does not explicitly state when to use this tool versus alternatives like get_verification or reconcile_verification. No when-not or exclusion criteria are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verify_refundA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations (readOnlyHint=false, openWorldHint=true, destructiveHint=false) are provided. The description adds behavioral context: it checks for duplication, mismatch, and policy failure, and returns a signed receipt. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loaded with the core purpose, and includes important behavioral details and output. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the verification logic and return values adequately given 10 parameters and no output schema. It could mention error handling or performance, but is sufficient for agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with detailed parameter descriptions. The overview description does not add much beyond the schema, but the schema itself is sufficient. Baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool verifies a refund against the system of record, listing specific checks (exists, amount, currency, customer, duplication) and outputs classification and signed receipt. It distinguishes from siblings like verify_cancellation by focusing on refunds.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description says to use when the agent claims to have issued a refund, providing clear context. It does not explicitly mention when not to use or name alternatives, but the purpose is clear enough for an agent to decide.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verified_completion_rateA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds value by defining the metric and breakdown, but the read-only behavior is already covered by annotations (readOnlyHint=true). No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, front-loaded with action, no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter read tool without an output schema, the description is sufficiently complete, covering what it reads and the breakdown.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist, so baseline 4 applies. The description does not need to add parameter details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Read') and resource ('Verified Completion Rate'), defines the metric, and mentions a breakdown of failures. It clearly distinguishes from sibling tools like get_verification which likely operate on individual items.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for retrieving an aggregate metric, but does not explicitly state when not to use or provide alternative tool names. However, it is clear enough given the sibling context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.