Server Quality Checklist

Tool Scores

• list_checksA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided; description implies a read operation but does not disclose side effects, permissions, or any behavioral traits beyond listing.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence with verb front-loaded, no wasted words. Very concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with no parameters and an output schema, the description is adequate. However, could briefly mention relationship to 'run_checks' for completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist; baseline is 4 as per guidelines. Description adds no parameter info but none is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description states verb 'List', resource 'declared data-quality checks', and scope 'the layer can run'. Clearly distinguishes from siblings like 'run_checks'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives such as 'run_checks' or other list tools. Lacks context for decision-making.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_metricA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full responsibility. It mentions 'pre-approved' (suggesting authorization) and 'fixed aggregation' (implying immutability), but does not explicitly state that the operation is read-only, nor does it detail potential side effects or permissions. The behavior is somewhat transparent but not fully.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences front-load the core action ('Compute a pre-approved metric') and then add constraints. Every word adds value; no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 4 parameters, no annotations, but an output schema exists, the description is too brief. It lacks details on parameter formats, output expectations, and prerequisites. It does not fully equip an agent to use the tool correctly without additional knowledge.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% (no parameter descriptions in schema), so the description must compensate. It mentions 'grouped/filtered by allowed dimensions', which loosely maps to dimensions and filters, but does not explain the 'metric' (required), 'limit' (ignored), or specifics of filters format. This is insufficient for a tool with 4 parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'compute' for a metric resource, and specifies optional grouping/filtering. It distinguishes itself from sibling tool 'list_metrics' (which just lists available metrics) by emphasizing computation with a fixed aggregation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool (to compute a pre-approved metric) and hints at constraints like 'allowed dimensions', but does not explicitly state when not to use it or provide alternative tool names. The context of sibling tools partially fills this gap.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• audit_verifyA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses the tool reports on call alterations and mentions the 'if enabled' condition, but lacks details on prerequisites (e.g., permissions), side effects, or whether it is read-only/destructive. This is adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary purpose, and contains no redundant or unnecessary words. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with no parameters and an output schema present, the description adequately explains the tool's action and result. It mentions the 'if enabled' precondition, which is important context. Could be slightly improved by noting what happens when audit is disabled, but not essential.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and schema coverage is 100% (trivial). The description adds no parameter information because none exist, which is appropriate. Baseline for 0 parameters is 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: verifying the tamper-evident audit chain and reporting alterations. It uses specific verbs ('verify', 'reports') and identifies the resource ('audit chain'). It is distinct from sibling tools, which involve describing entities, getting metrics, or running checks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when audit is enabled but does not explicitly state when to use this tool versus alternatives. No exclusions or comparisons to siblings are provided, leaving the agent to infer context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_checksA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses that each check is read-only ('read-only violation count'), the return structure (readiness score, overall status, per-check breakdown), and the severity mapping for status ('error' -> failing, 'warn' -> degraded). Since there are no annotations, the description fully bears the transparency burden and does so well.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (three sentences) and front-loaded with the main purpose. Every sentence adds value: main action, read-only guarantee, return structure, and status logic. No redundant or vague phrasing.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters and an output schema present, the description thoroughly explains what the tool does and what it returns. It covers the key behavioral aspects (read-only, severity, output format) without relying on the output schema. It is complete for this tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero parameters, so schema coverage is 100%. Per guidelines, zero parameters merits a baseline of 4. The description does not need to add parameter information, and it doesn't.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool runs data-quality checks and returns a readiness summary. It specifies the verb 'Run' and the resource 'declared data-quality checks', which distinguishes it from sibling tools like 'list_checks' (which lists checks) and 'audit_verify' (likely a different type of check).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not explicitly state when to use this tool versus alternatives. While the purpose is clear, it lacks direct guidance on prerequisites, when to prefer it, or when to avoid it. It is adequate but not proactive in helping the agent choose among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.