Server Quality Checklist

Tool Scores

• agent_invariants_summarize_traceC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavioral traits. It only says 'deterministic' but omits side effects (likely read-only), required permissions, or limitations. Minimal transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, no wasted words. However, it lacks critical information needed for proper use, making it under-specified rather than efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The complex input schema (over 20 fields with nested objects) is not explained, nor is the output format. No output schema, so the description should cover return structure but does not.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, and the description does not explain the 'events' parameter or its structure. No hint about the expected array of event objects with specific fields.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states it returns counts for tools, approvals, failures, completion claims, and observed outcomes, which clarifies the tool's aggregating function. However, the verb 'summarize' is missing from the description, and 'deterministic counts' is slightly vague. It distinguishes from siblings like validate_contract or check_trace.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like agent_invariants_check_trace or agent_invariants_compare_traces. The description does not mention context or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• agent_invariants_check_traceB
  3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It mentions the types of rules evaluated (deterministic, etc.), giving some behavioral insight. However, it does not disclose side effects, read-only nature, auth requirements, or error behavior. The list of rule types is helpful but incomplete.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that front-loads the action and resource. It is free of redundancy, but lacks structure (e.g., headers or bullet points) that would improve scanability. Efficient but not perfectly structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the high complexity (nested objects, 11 rule kinds, no output schema), the description is insufficient. It does not explain input interpretation, return format, error handling, or how results are structured. The agent would need supplementary knowledge to use this tool effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description adds no information about the two required parameters ('events' and 'contract'). Both are complex nested objects, but the description does not hint at their structure or purpose, leaving the agent to rely solely on the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Evaluate' and the resource 'normalized agent event trace' against specific rule categories (permission, stop, completion, retry, scope, ordering). This distinguishes it from sibling tools like validate_contract (likely validates contract structure) and summarize_trace (summarizes).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. There is no explicit 'when to use' or 'when not to use' information, nor mention of preconditions or caveats. Usage is only implied by the tool name and context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• agent_invariants_compare_tracesB
  3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided; description carries full burden. It reports outputs but does not disclose whether the tool is read-only, requires specific permissions, or any side effects. For a comparison tool, it likely is read-only, but this is not stated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence with clear structure: action and output summary. Could be slightly improved by noting required parameters, but no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (3 complex parameters, no output schema, no annotations, and sibling tools), the description is insufficient. It lacks explanation of the contract parameter, trace structure, and report format, making it hard for an agent to use correctly without prior knowledge.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%. The description only mentions 'baseline and candidate traces', ignoring the 'contract' parameter entirely. No additional meaning beyond parameter names is provided, leaving the agent to guess the structure and format of these complex nested objects.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the verb 'Compare' and the objects 'baseline and candidate traces', and lists specific report outputs ('newly broken rules, new tools, call inflation, or outcome regression'). Differentiates from sibling tools like validate_contract or check_trace by focusing on comparison.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use (when comparing two traces) but provides no explicit when-not-to or alternatives. No guidance on avoiding this tool in favor of siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• agent_invariants_validate_contractA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, placing full burden on description. The description mentions specific validation aspects (unique IDs, bounded matchers) but omits critical details: what happens on validation failure (e.g., errors returned), whether the tool is idempotent or modifies state, or any permissions required.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, well-crafted sentence of 12 words with no redundancy. The verb 'validate' and object 'contract' are front-loaded, making the purpose immediately clear.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complex nested input schema, 0% parameter descriptions, and no output schema or annotations, the description is too terse. It does not explain return format, error handling, version constraint (v1 only), or the meaning of 'bounded matchers.' Sibling tools provide no additional context for this specific tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage for the parameter's sub-properties is 0%, so description must compensate. It adds context by naming what is validated (unique rule IDs, bounded matchers), which goes beyond the raw schema. However, it could detail the parameter structure more explicitly.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool validates a version 1 Agent Invariants contract, specifying 'unique rule IDs and bounded matchers' as key validation foci. This distinguishes it from sibling tools (check_trace, compare_traces, summarize_trace) that operate on traces, not contracts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives. It is implied that validation should precede trace analysis, but no when-not-to-use or prerequisites are mentioned. The sibling tools' purposes are different, but the description does not clarify when validation is needed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.