Server Quality Checklist

Tool Scores

• evaluate_actionA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries burden. It discloses return decisions and parameter roles but lacks details on side effects, authentication, rate limits, or error handling. The description adds moderate transparency beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the main purpose, followed by parameter explanations. It is reasonably concise without wasting words, though the bullet-style list could be more compact.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema existing, return values are not fully described, but the description mentions three decisions. It lacks examples or constraints on parameter values. For a tool with 5 params and 0% schema coverage, the description is adequate but not deeply complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description compensates by explaining each parameter's meaning (action_type, target, params, description, agent_id). While not exhaustive, it provides sufficient context for an agent to understand required inputs.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool evaluates an AI agent action against governance policy and returns a decision (auto, approve, block). It uses a specific verb-resource combination and distinguishes itself from siblings by focusing on individual action evaluation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus siblings like check_risk or evaluate_batch. No explicit when-to-use or when-not-to-use conditions are provided, leaving the agent to infer context independently.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• evaluate_batchA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full responsibility for behavioral disclosure. It mentions 'evaluate' which suggests a read operation, but does not explicitly state whether it is read-only, whether it modifies state, or any side effects. Missing details on permissions, rate limits, or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and includes a one-line summary followed by a bullet-like list of required fields. It could be more structured (e.g., using proper Args format), but it avoids unnecessary verbosity and is easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  While the output schema exists and may cover return values, the description does not mention what the tool returns (e.g., per-action results) or error handling. For a tool with one parameter, it covers input adequately but not output or edge cases. Some completeness is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides no parameter descriptions (0% coverage), so the description adds value by specifying that each action dict should contain 'action_type', 'target', and optionally 'params', 'description', 'agent_id'. However, it does not define the types or expected formats for these fields, leaving ambiguity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Evaluate multiple actions at once against the governance policy' and distinguishes from the sibling 'evaluate_action' by emphasizing batch processing. It also lists the required fields for each action dict, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It implicitly advises using this tool for batch evaluation of multiple actions rather than single-action evaluation, but does not explicitly contrast with alternatives like 'evaluate_action' or state when not to use it. Could be more explicit about the batch vs. single distinction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_riskA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses that the tool returns risk level and approval requirement and is lighter than evaluate_action. With no annotations provided, it moderately compensates by indicating the tool is read-like and partial, but lacks details on authentication needs, rate limits, or side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, consisting of two sentences and a bulleted list of arguments. The first sentence clearly states the purpose, and every subsequent sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 parameters, no enums) and the presence of an output schema (though not shown), the description adequately explains what it does and what it returns. It lacks only minor details about parameter constraints or advanced behavior, but is complete enough for correct usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, but the tool's description elaborates on the parameters: action_type is 'the kind of operation' and target is 'the system being acted upon.' This adds meaningful context beyond the schema titles, though it could be more precise with examples or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs a 'Quick risk check for an action type + target combination' and explicitly differentiates itself from the sibling evaluate_action by noting it is 'lighter'. This provides a specific verb and resource, making its purpose unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description indicates this tool is 'lighter than evaluate_action' and returns only risk level and approval requirement, implying it should be used for quick checks. However, it does not explicitly state when not to use it or mention any prerequisites or alternatives beyond evaluate_action.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_policyA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It accurately describes a read-only retrieval operation without misleading claims. Could mention caching or latency but not necessary.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences that are front-loaded with the purpose and then detail. Every sentence adds value; no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Tool has no parameters and has an output schema. Description explains what the output contains. For a simple retrieval tool, this is complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters defined; schema coverage is 100%. Description adds no parameter detail because none exist, which is appropriate. Baseline 4 per rule.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Get the current governance policy rules' and enumerates what is returned (action patterns, targets, risk levels, approval requirements). It distinguishes from siblings like update_policy.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use vs. alternatives (e.g., check_risk). Implied by name, but lacking explicit when-not or context for choosing this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• update_policyA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description states the effect: 'takes effect immediately for all subsequent evaluations,' which provides behavioral transparency. Since no annotations are provided, the description carries the full burden, and it adequately conveys the impact.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two sentences and an Args section, front-loaded with the main action. No unnecessary information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that an output schema exists and the tool is simple, the description covers the input and effect adequately. It could mention error handling or validation but is mostly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description includes an Args section that explains the yaml_content parameter as 'YAML string containing the policy rules,' adding meaning beyond the schema, which has 0% description coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Hot-reload the governance policy from a YAML string.' It specifies the verb (hot-reload), resource (governance policy), and input (YAML string), distinguishing it from sibling tools like check_risk, evaluate_action, evaluate_batch, and get_policy.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions that the new policy takes effect immediately, implying real-time usage. However, it does not provide explicit guidance on when to use this tool versus alternatives or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.