Server Quality Checklist

Tool Scores

• list_templatesC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It states the action but doesn't describe what 'list' entails (e.g., format, pagination, sorting), whether it's read-only or has side effects, or any constraints like rate limits or authentication needs. This leaves significant gaps for a tool that likely returns data.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero wasted words. It's front-loaded with the core action and resource, making it easy to parse quickly, which is ideal for conciseness in tool descriptions.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no annotations, no output schema, and low schema coverage, the description is incomplete. It doesn't explain return values, behavioral traits, or parameter usage, leaving the agent with insufficient information to use the tool effectively beyond its basic purpose.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 1 parameter with 0% description coverage, so the description must compensate. It mentions 'available policy templates' but doesn't explain the optional 'category' parameter or how filtering works. This adds minimal value beyond the schema's structure, failing to address the coverage gap adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('available policy templates'), making the purpose immediately understandable. However, it doesn't differentiate this tool from potential sibling operations like 'generate_policy' or 'test_policy' beyond the basic action, which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'generate_policy' or 'validate_policy'. There's no mention of prerequisites, typical use cases, or exclusions, leaving the agent to infer usage context entirely from the tool name.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_policyC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the validation action but fails to describe key traits such as whether it's read-only or destructive, what the output format is (e.g., success/failure, error details), or any rate limits or authentication needs. This leaves significant gaps for a tool that likely returns validation results.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without any fluff. It is appropriately sized and front-loaded, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a validation tool with no annotations, no output schema, and low parameter coverage, the description is incomplete. It lacks information on behavioral traits, parameter details, and expected outputs, which are crucial for an agent to use the tool effectively in context with siblings like test_policy.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description must compensate for the undocumented parameter. It only implies that 'policy_yaml' is related to policy syntax validation but adds no details about the parameter's format, constraints, or examples (e.g., YAML structure, required fields). This is insufficient given the low coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as validating policy syntax using a specific method (cerbos compile). It specifies the verb 'validate' and the resource 'policy syntax', though it doesn't explicitly differentiate from siblings like test_policy or suggest_improvements, which might have overlapping validation aspects.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives such as test_policy or suggest_improvements. It lacks any context about prerequisites, typical use cases, or exclusions, leaving the agent to infer usage from the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• suggest_improvementsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions analysis and suggestions but doesn't describe what the analysis entails, how suggestions are generated, whether it's read-only or has side effects, or any performance/rate limit considerations. This leaves significant gaps for a tool that analyzes security policies.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that gets straight to the point without unnecessary words. It's appropriately sized for a tool with 2 parameters, though it could be more structured with separate purpose and usage sections.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a security analysis tool with 2 parameters (one complex nested object), no annotations, and no output schema, the description is inadequate. It doesn't explain what security issues are analyzed, what format improvements are suggested in, or how the optional 'icp' enhances analysis. More context is needed given the complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 50% (only 'icp' has a description), so the description should compensate but doesn't. It mentions 'policy' analysis but doesn't explain what 'policy_yaml' should contain or what 'icp' represents beyond the schema's 'Optional ICP for enhanced analysis'. The description adds minimal value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('analyze', 'suggest') and resource ('policy for security issues'), making it easy to understand what it does. However, it doesn't differentiate from sibling tools like 'validate_policy' or 'test_policy', which likely have overlapping security analysis functions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'validate_policy' or 'test_policy'. It doesn't mention prerequisites, exclusions, or specific contexts where this analysis is preferred over other policy-related tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• test_policyC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'Run cerbos test' but doesn't explain what this entails—e.g., whether it's a read-only operation, if it modifies data, what the output looks like, or any error conditions. This leaves significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste. It's front-loaded with the core action and target, making it easy to parse quickly without unnecessary details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of testing policies (which could involve validation, execution, or reporting), the description is incomplete. With no annotations and no output schema, it fails to explain what the tool returns (e.g., test results, errors) or behavioral traits like side effects. This makes it inadequate for an agent to use the tool effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, clearly documenting both parameters as YAML content for policy and test suite. The description adds no additional meaning beyond this, as it doesn't elaborate on parameter formats, constraints, or examples. Baseline 3 is appropriate since the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Run cerbos test') and the target ('on policy and test suite'), which is specific and understandable. However, it doesn't explicitly differentiate this tool from sibling tools like 'validate_policy' or 'suggest_improvements', which might have overlapping purposes in the Cerbos policy testing domain.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'validate_policy' or 'suggest_improvements'. It lacks context about prerequisites, such as needing valid YAML content, or exclusions, leaving the agent to infer usage based on the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_policyA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but only states what the tool does, not how it behaves. It doesn't disclose error handling, rate limits, authentication needs, or what 'enterprise-grade' entails. The description adds minimal behavioral context beyond the basic function.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, zero waste, front-loaded with the core purpose. Every word earns its place without redundancy or unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 2 parameters, 100% schema coverage, but no annotations or output schema, the description is adequate but has gaps. It explains the transformation purpose but lacks details on behavioral traits, error cases, or output format, which would be helpful given the complexity of policy generation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so parameters are documented in the schema. The description doesn't add meaning beyond what the schema provides about 'nl_requirements' or 'icp'. Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verb ('Convert') and resource ('natural language guardrails into enterprise-grade Cerbos YAML policies'). It distinguishes itself from siblings by focusing on generation rather than listing, suggesting, testing, or validating policies.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for converting natural language to YAML policies, but doesn't explicitly state when to use this vs. alternatives like 'suggest_improvements' or 'test_policy'. No guidance on prerequisites or exclusions is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.