Server Quality Checklist

Tool Scores

• find_swallowed_exceptionsA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It clearly states the tool scans code and outputs findings with line number, severity, and code excerpt, indicating a read-only, non-destructive operation. It could mention no side effects explicitly, but is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no wasted words: the first explains purpose and patterns, the second describes output. Every part adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter tool with full schema coverage and no output schema, the description sufficiently covers what the tool does and what it returns. It could explicitly state the return format (e.g., list of objects), but it is implied.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds context about the patterns detected, which helps understand how the 'code' parameter is used, but does not add format or constraints beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Scan' and resource 'Python source code', and explicitly lists the patterns it detects (pass-only handlers, mock-substitution returns, etc.), clearly differentiating it from sibling tools that deal with transcripts or verification.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for detecting specific code smells but does not specify when to use this tool versus alternatives or mention any exclusions. Context from sibling names shows distinct use cases, but explicit guidance is missing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• review_transcriptA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It discloses the tool's behavior: it flags specific types of issues (unverified claims, contradictions, tool calls without side effects). No mention of destructive actions or auth needs, but for a review tool this is acceptable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with key purpose and specific detection criteria. No redundant or unnecessary words. Efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has one parameter, no output schema. Description explains input and detection criteria well, but could mention the output format (e.g., list of issues) for completeness. Still adequate for the complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with descriptions for the transcript items. The description adds value by explaining the structure and purpose of the parameter ('Pass an array of {role, text, tool_calls?} objects'). It clarifies the expected input beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool's purpose: reviewing multi-turn agent transcripts for specific issues (unverified claims, contradictions, tool calls without side effects). It distinguishes from sibling tools like find_swallowed_exceptions and verify_action_outcome by focusing on transcript analysis.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Description implies usage for analyzing agent transcripts and lists what it detects, but does not explicitly state when to use this tool versus alternatives or when not to use it. The context of sibling tools helps, but explicit guidance would improve.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verify_action_outcomeA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavior: pure function, stateless, returns structured verdict. It details the verdict categories and limitation of snapshot capture, providing complete transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is detailed and informative, but somewhat verbose. It front-loads the purpose but includes extensive examples and context that could be condensed without losing value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (4 params, nested objects, no output schema), the description is highly complete. It explains the return type, verdict values, and use cases, leaving no major gaps for an agent to understand usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and schema descriptions are present. The description adds valuable context (examples, recognized keys, formats) that goes beyond the schema, aiding parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool compares agent's stated outcome against before/after snapshots to detect misreporting, with specific failure modes and a clear verdict output. It distinguishes effectively from sibling tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use the tool and provides context (pure function, stateless), but does not explicitly state when not to use it or offer direct comparisons to sibling tools, which slightly reduces clarity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verify_response_groundingA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries the full burden. It discloses that the tool is sub-second, local, no API key, returns per-claim and overall verdict, and notes a quirk about the question parameter (stored but unused in v1.0). It lacks details on error handling or edge cases but provides substantial behavioral insight.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, each earning its place: first defines action and output, second gives usage context, third adds performance and privacy traits. No fluff, no repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description explains return format (per-claim + verdict). It covers usage, parameters, behavior, and performance. For a verification tool with simple inputs and outputs, this is fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, and the description adds meaningful context beyond the schema: it explains the threshold parameter (Jaccard overlap, default 0.30, effect of lower/higher values) and the question parameter's current behavior (stored but unused). This fully compensates for any schema gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Check' and resource 'every claim in answer has support in context'. It explicitly names the return values (per-claim grounded/ungrounded + overall verdict) and distinguishes itself from sibling tools like find_swallowed_exceptions or review_transcript by focusing on factual grounding.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description advises to 'Use inline during an agent conversation to flag hallucinated responses before they become user-facing facts'. This gives a clear when-to-use scenario. It does not explicitly list when not to use or alternatives, but the context is sufficient for an agent to understand its purpose.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.