Server Quality Checklist

Tool Scores

• evaluate_permission_toolC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of disclosing side effects, permissions, or error behavior. It only mentions return values ('decision, matched rules, and risk score'), which are likely covered by the output schema. It does not state whether the operation is read-only, whether it modifies state, or what happens with invalid input.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded with the main purpose, followed by return-value summary. It avoids unnecessary words and is well-structured. However, it could be improved by integrating parameter details without becoming verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having an output schema and only two parameters, the tool involves nested objects and no annotations. The description does not cover prerequisites, edge cases, or input formatting, making it insufficient for an agent to confidently invoke the tool in all situations. It is more complex than a simple filtered-list tool and requires more context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, yet the description does not explain the meaning or structure of the 'action' or 'policy_rules' parameters. It briefly references 'policy rules' but fails to define expected fields, formats, or examples, leaving the agent without enough information to construct valid inputs.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Evaluate') and resource ('whether an agent action is allowed under the given policy rules'). It distinguishes itself from siblings by focusing on permission evaluation, whereas siblings address failure detection, reliability scoring, and audit reports.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The description implies a use case for policy compliance, but it does not explicitly mention exclusions, prerequisites, or refer to sibling tools for different scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• detect_silent_failures_toolB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry full behavioral disclosure. It states the detection condition but does not reveal whether the tool is read-only, what exact output it produces beyond flagging, whether it requires special permissions, or how it handles edge cases (e.g., logs with error flags). The description provides only a minimal behavioral snapshot.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that is front-loaded with the action and outcome. It contains no filler or redundant information, making it maximally concise while still conveying the core purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has a low complexity (one parameter, no nested objects) and an output schema, so return value details are not needed. However, the description omits crucial context about the input format, usage scenarios relative to sibling tools, and any behavioral caveats. It is adequate only for a superficial understanding, but not complete for an agent to invoke it correctly without additional external knowledge.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and there is only one parameter ('logs') with no description of its expected structure or content. The description implies the tool consumes logs, but it does not explain what fields or formatting are required, nor does it compensate for the schema's lack of detail.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Flag') and identifies the resource ('tool calls') with a clear condition ('returned empty or null content without an error flag'). It fully distinguishes this from sibling tools, which focus on scoring reliability, evaluating permissions, and generating audit reports.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool (when there is a concern about empty/null results without errors) but does not explicitly state alternatives or exclusionary conditions. No mention of when not to use it or how it relates to sibling tools, leaving the context implicit rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• score_agent_reliability_toolA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must convey behavioral traits. It clarifies input requirements (log entry fields) but does not describe edge-case behavior, computation details, or handling of malformed data. As a read-only computation tool, it omits side-effect disclosures that would be needed for mutating tools.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the tool's purpose, then a compact list of expected fields. Every sentence earns its place without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The presence of an output schema covers return values, but the description still leaves gaps: no mention of how weights affect scoring, no guidance on edge cases like empty logs or missing fields, and no interpretation of the reliability score. Enough for a simple tool, but missing key usage context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage. The description adds valuable semantics for the 'logs' array structure (expected fields like tool_name, success, latency_ms), but it does not explain the 'weights' parameter at all, leaving that parameter underspecified.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Compute overall reliability score (0-100)') with a clear resource ('from tool-call logs') and output range. This distinguishes it from sibling tools like detect_silent_failures or generate_audit_report, which focus on different aspects.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage: when you have tool-call logs and need a reliability score. However, it does not explicitly mention when not to use this tool or suggest alternatives among siblings, leaving some room for confusion in edge cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_audit_report_toolA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It mentions the output is 'markdown governance/audit report' and that it uses logs and optional scores, but it does not disclose side effects, permissions, or any specific behaviors beyond generation. Since generation is inherently non-destructive, this is a minor gap; however, additional context about the report's contents or how scores are integrated would improve transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that directly states the action and resource. There is no wasted text, and it conveys all essential information efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple report-generation tool with an output schema, the description covers the key inputs and output format. It could be more complete by referencing the sibling tools as pipeline steps or clarifying the report's scope, but given its simplicity and the presence of an output schema, it's sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It does by naming both parameters: 'logs' (the data source) and 'pre-computed scores' (optional input). This clarifies that 'logs' is the primary input and 'scores' is an optional enhancement, adding meaning beyond the bare schema definitions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Generate a markdown governance/audit report from logs (and optional pre-computed scores).' It uses a specific verb ('Generate') and a specific resource ('markdown governance/audit report'), and it distinguishes itself from sibling tools by focusing on report generation rather than detection, scoring, or evaluation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is used when you have logs (and optionally pre-computed scores) and need a governance/audit report. While it doesn't explicitly mention alternatives or exclusions, the context of sibling tools (detect, score, evaluate) makes it clear this is the report-generation step. The phrase 'and optional pre-computed scores' hints at a preceding scoring step, providing useful context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.