Server Quality Checklist

Tool Scores

• preclear_rule_requestB
  3.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the behavioral disclosure burden. It usefully discloses that the operation is read-only and deterministic, which signals safety and reproducibility. However, it does not describe what the checks actually validate, what failure/success looks like, or any side effects or prerequisites, leaving meaningful gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, tightly written sentence with no filler. Key qualifiers ('deterministic', 'read-only') are front-loaded, but the sentence is slightly jargon-heavy and could benefit from a brief expansion to define 'preclear checks' without becoming verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the opaque required request object, three undocumented parameters, and absence of annotations, the description is not sufficient for correct invocation. An output schema may exist, but it does not make up for missing input semantics, so the overall context remains incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not compensate. The required 'request' object, 'rulebase', and 'device_group' parameters are left entirely unexplained, so an agent cannot determine what content the request object should contain or how the optional parameters influence the check.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Run ... preclear checks') and a clear resource/timing ('before a rule change is considered'). It distinguishes the tool from the get/list/search siblings by framing it as a pre-change validation step. The term 'preclear' echoes the tool name, but the added 'deterministic read-only' qualifier clarifies the nature of the operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives an explicit trigger context: use this before a rule change is considered. This is clearer than merely implying usage, but it does not enumerate alternatives or explicitly say when not to use it relative to the sibling tools, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_security_rulesB
  3.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries behavioral weight: 'Fetch' implies a read operation, 'normalized' indicates output transformation, and 'from one...' implies a single scope. It does not disclose error handling, pagination, permissions, or null-parameter behavior, but the verb and scope give basic behavioral orientation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single front-loaded sentence with no filler; every word adds meaning. It is appropriately sized for the tool's simple scope.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Although an output schema exists, the description leaves key contextual gaps: no parameter semantics, no behavior when both optional fields are omitted, and no routing to sibling tools. For a tool with zero annotations and 0% schema coverage, this is not complete enough for confident invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description only repeats the fields' names (device group, rulebase) without explaining allowed values, formats, defaults, or null behavior. It adds slight meaning by saying 'from one', but it does not compensate for the undocumented schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Fetch'), a concrete resource ('normalized security rules'), and a scope ('one Panorama device group and rulebase'), making the operation clear. It distinguishes this from sibling tools like list_device_groups and preclear_rule_request through the object and scope, though it does not name alternatives explicitly.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when an agent needs normalized security rules scoped to a single Panorama device group and rulebase. However, it does not explicitly contrast this with siblings such as search_policy_coverage or state when not to use it, so the guidance is implied rather than direct.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_policy_coverageA
  3.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. 'Without changing Panorama' is a clear, essential non-mutation guarantee, which is especially valuable given the search/analysis nature of the tool. It does not disclose permissions or performance characteristics, but the most critical safety trait is present.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The entire description is a single front-loaded sentence: it begins with the action, specifies the scope (duplicates, coverage, conflicts, overlaps), and appends the key constraint (no mutation). There is no filler, repetition, or wasted wording.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has a complex, required request object and no parameter descriptions, yet the description does not explain what goes in that object or how rulebase/device_group affect the search. The output schema may document return values, but invocation semantics are far too thin for an agent to call this tool correctly without additional inference.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description provides no guidance on the three parameters (request, rulebase, device_group). The required 'request' object is completely opaque, with additionalProperties true and zero explanation, so an agent has no idea how to construct a valid invocation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Find') and names concrete resources (exact duplicates, coverage, conflicts, partial overlaps), while adding a non-mutation guarantee. This clearly differentiates it from siblings like get_security_rules, which retrieve rules, and preclear_rule_request, which likely modifies state.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this is a policy-analysis tool but does not explicitly state when to prefer it over siblings or list exclusions. An agent can infer the use case from the verb and resources, but there is no direct routing guidance like 'use instead of get_security_rules when assessing overlap.'

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_device_groupsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that the operation is read-only and visibility-limited, which is valuable behavioral context. It does not cover edge cases like pagination or empty results, but that is minor for a parameterless listing tool with an output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no filler or redundancy. It states the operation, resource, and scope efficiently, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a parameterless read-only list tool with an output schema, this description is complete. It identifies what is listed, the resource context, and the access boundary enforced by the read-only API key, leaving no critical information missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the schema provides no parameter semantics. The description adds useful context by defining the scope of the listing through the API key's visibility, which satisfies the baseline for parameterless tools.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') and names the exact resource ('Panorama device groups'), while clarifying the scope ('visible to the configured read-only API key'). This distinguishes it clearly from sibling tools like get_security_rules and search_policy_coverage.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool—when listing device groups visible to the configured API key—but does not explicitly state when not to use it or mention alternatives. Usage context is clear enough for a simple read-only listing, but there is no direct guidance about sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.