Server Quality Checklist

Tool Scores

• parse_policyA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description carries full burden. It describes the parse operation and output format, but does not disclose whether the tool is read-only or has side effects. The behavior is straightforward for a parsing tool, but lacks details like error handling or idempotency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences front-load the core functionality and immediately provide context for usage. Every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity and the presence of an output schema, the description is adequate. It explains the purpose and how the output ties into analysis, though it could mention potential limitations like config size or parser constraints.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents parameters. The description adds value by explaining the output's relationship to another tool, but does not provide additional parameter-level details beyond what the schema offers.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Parse' and the resource 'vendor-native firewall config', with the outcome 'return normalized JSON rules'. It distinguishes from sibling tools by explicitly mentioning 'before running overlap analysis', referencing the analyzer tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit guidance: 'Use this to inspect ... before running overlap analysis.' This indicates when to use. It does not explicitly state when not to use, but the context implies it is for verification, not analysis. Sibling tool names provide further differentiation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_firewall_rule_overlapA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations. Description conveys non-destructive analysis and detection capabilities, but does not explicitly state idempotency or side-effect absence. Output schema exists, reducing need for return value details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences: first states purpose, second lists detections, third explains modes and workflow. No wasted words, front-loaded with essential info.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Explains prerequisite (use parse_policy) and input modes adequately. Omits mention of optional parameters but schema handles that. Output schema present so return values are covered.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%; description adds value by grouping parameters into two modes (vendor-native vs pre-normalized), which aids understanding beyond individual parameter descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'analyze' and the resource 'firewall rule overlap', enumerating detection types like duplicates and shadowed rules. It distinguishes from siblings: batch_analyze_overlap is batch, parse_policy is preprocessing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly recommends using parse_policy before analysis and describes two input modes. Lacks explicit when-not-to-use guidance, but the modes imply appropriate contexts.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• batch_analyze_overlapA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It reveals that existing_rules is parsed once and reused (a behavioral optimization), but does not disclose error handling, limitations, or whether the tool is read-only. More transparency on edge cases would improve the score.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description consists of two concise, front-loaded sentences with no redundancy. Every sentence adds value: purpose, efficiency comparison, and prerequisite workflow.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has two well-described parameters, an output schema exists, and siblings are clearly listed, the description provides sufficient context: it explains the advantage over the sibling, the prerequisite step using parse_policy, and the usage pattern, making it complete for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with detailed parameter descriptions already explaining the structure and reuse behavior. The tool description reinforces the workflow but adds little new semantic insight beyond what the schema already provides, keeping the score at baseline 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool analyzes multiple candidate rules against an existing ruleset in a single call, distinguishing it from the sibling analyze_firewall_rule_overlap by highlighting the efficiency of reusing parsed existing rules.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly recommends using parse_policy first to obtain normalized rules, advises to pass candidates all at once, and positions this tool as more efficient than calling the sibling multiple times, providing clear when-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_supported_vendorsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full disclosure burden. It correctly indicates a read-only listing operation and hints at the returned content (vendor identifiers and format requirements). While it does not detail output schema or performance traits, the tool is simple and zero-parameter, so the description is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences: the first describes the action, the second provides usage context. No extraneous words, and critical information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (no parameters, output schema exists), the description fully covers its purpose and context. It links to sibling tools, making clear how it fits into a larger workflow.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so schema coverage is 100%. The description does not need to add parameter details. The baseline for no parameters is 4, and the description meets this by not requiring further clarification.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') and resource ('supported firewall vendors and their configuration format requirements'). The purpose is unambiguous and distinct from sibling tools like analyze_firewall_rule_overlap, which consume this information.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool: 'Use this to understand what vendor identifiers and payload formats are accepted by analyze_firewall_rule_overlap and parse_policy.' This provides clear context and guidance on the tool's role in a workflow.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.