Server Quality Checklist

Tool Scores

• assay_explain_traceC
  2.2/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No behavioral traits disclosed; no annotations provided. Description does not mention return value, side effects, or permissions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is very concise (single sentence) but lacks structure; could benefit from additional context without being verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 3 parameters, no output schema, and no annotations, the description is severely incomplete for an AI agent to use correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but description adds no details about parameters; only implies policy and trace are inputs without explaining their roles or the format parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool explains trace evaluation against a policy, distinguishing it from sibling tools focused on checking or deciding.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like assay_policy_decide or assay_check_sequence.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• assay_check_sequenceC
  2.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It does not mention side effects, permissions, rate limits, or any constraints beyond the stated purpose.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is only one sentence, which is too short to provide adequate information for a tool with 3 parameters and no other documentation. It is under-specified, not appropriately concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, no annotations, and a complex task (validation logic), the description is completely inadequate. It does not explain what 'allowed' means, what the parameters do, or what the tool returns.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 3 parameters (history, next_tool, policy) but the description mentions none of them. Schema coverage is 0%, so the description should compensate, but it does not.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Validate if a tool call is allowed given the history' clearly states the verb 'validate' and the resource 'tool call allowed'. However, it does not differentiate from sibling tools like assay_check_args, which likely validates arguments, so it is not a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives, no explicit context or exclusions. It only states the general purpose.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• assay_check_coverageC
  2.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations and no output schema, the description fully carries the burden of behavioral disclosure. It does not mention side effects, permissions, return format, or any behavioral traits. The word 'analyze' implies read-only, but this is not explicitly stated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, which is concise. However, it is under-specified for a tool with 4 parameters; it trades substance for brevity. While efficient in word count, it lacks the necessary structure and detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 4 parameters, 2 required, no output schema, and no annotations, the description is completely inadequate. It does not explain parameters, return values, or behavioral context, making it impossible for an agent to use the tool effectively without additional knowledge.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must explain what each parameter means. The description only provides the tool's high-level purpose and gives no information about 'policy', 'traces', 'threshold', or 'format'. This leaves the agent with no understanding of how to correctly fill the parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('analyze') and the resource ('trace coverage against a policy'), which is specific enough to understand the tool's main function. However, it does not differentiate from sibling tools like assay_check_args or assay_check_sequence, which also check aspects of assay policies.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives such as assay_check_args or assay_check_sequence. Without context on coverage vs. args/sequence, an agent cannot decide which tool is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• assay_policy_decideC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries full burden but only states it checks blocking status. It fails to disclose whether the tool modifies state, requires authentication, or has side effects, leaving behavioral ambiguity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence, but it is too sparse to be fully effective; it earns high marks for brevity but loses a point for sacrificing necessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the absence of output schema and annotations, the description is insufficient for an agent to understand return values, error conditions, or interpretation of 'blocked' status, making it incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not explain what 'tool' or 'policy' parameters represent, leaving the agent with no semantic guidance for correct invocation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Check' and identifies the resource 'if a tool is blocked by policy', clearly differentiating from sibling tools that check arguments, coverage, sequence, or traces.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No usage guidance is provided; the description does not indicate when to use this tool over siblings or mention any prerequisites or context for its use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• assay_check_argsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It merely states 'validate tool arguments' without any details on side effects, return behavior on success/failure, authentication needs, or rate limits. Since the tool performs validation, the description should at least hint at whether it returns a boolean, throws errors, or has any side effects. The current description lacks this essential transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise, consisting of a single sentence. While this is efficient and front-loaded, it may be too terse for a tool with three parameters (including a nested object) and no output schema. The description earns its place by being clear about the overall action, but it could be slightly expanded to include essential details without becoming verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (three parameters, nested object, no annotations, no output schema), the description is incomplete. It fails to describe the output/return value, which is critical for an agent to know how to use the tool. Additionally, it does not explain the relationship between parameters or provide any behavioral context. The description leaves significant gaps that could hinder correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, meaning the schema provides no parameter documentation. The description adds minimal context: it mentions 'tool arguments' and 'policy schema', implying that 'arguments' is the object to validate and 'policy' is the schema to validate against. However, it does not explain the 'tool' parameter or provide details on the structure of 'arguments' or 'policy'. Given the low coverage, the description should compensate with more detailed parameter semantics, but it does not.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Validate tool arguments against a policy schema.' It uses a specific verb ('validate') and resource ('tool arguments'). This clearly distinguishes it from sibling tools like assay_check_coverage, which check coverage, and assay_check_sequence, which check sequences. The purpose is unambiguous and well-defined.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not specify prerequisites, when validation is needed, or how it fits into a workflow. For example, it doesn't say whether this should be called before invoking another tool or as a standalone check. This omission leaves the agent without context for appropriate invocation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.