Server Quality Checklist

Tool Scores

• check_config_complianceA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It usefully discloses that there are ten deterministic checks, each returning a framework reference and a remediation line, and it lists the concrete checks. It does not explicitly state whether the operation is read-only, offline, or if any network/device access is required, though 'over one device config' implies offline analysis.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single dense sentence that front-loads the tool's purpose, then efficiently lists checks, output characteristics, and supported vendors. Every clause adds useful information with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The output schema covers return values, so the description does not need to explain those. What is missing is explicit treatment of the `vendor` parameter and clearer routing guidance versus siblings. For a simple two-parameter tool, the description is otherwise quite complete, but those gaps prevent a higher score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It makes clear that `config` is a device config and that supported vendors are Cisco, MikroTik, EdgeOS, and OPNsense. However, it never explicitly ties these vendors to the `vendor` parameter, does not state that `vendor` is optional or what happens when it is empty, and provides no parameter-level guidance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Run') and a specific resource ('CIS/PCI starter pack over one device config'), and enumerates the exact check categories. It clearly distinguishes this tool from siblings like sanitize_config or explain_config_diff by defining a distinct compliance-checking behavior.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage: it operates over one device config and supports four named vendors, so an agent can infer when to call it. However, it gives no explicit 'when not to use' or alternatives, such as pointing to sanitize_config for cleanup or explain_config_diff for change understanding.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lookup_ise_failure_codeA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description carries the behavioral disclosure burden. It usefully states the output content ('what it means, the usual cause, and the fix'), but it does not explicitly confirm this is a read-only lookup or describe behavior for unknown or malformed codes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single front-loaded sentence that names the action, resource, example inputs, and output structure. The list of twelve example codes adds length but is genuinely useful for confirming the expected input domain.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a one-parameter lookup tool with an output schema, the description is complete enough for an agent to select and invoke it correctly. The main gap is the absence of routing guidance relative to sibling diagnostic tools, but that does not prevent correct use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides only a bare string parameter with 0% description coverage, so the description must compensate. It adds a list of example codes and clarifies the parameter is a 'failure or step code', giving the agent a practical expectation. It stops short of specifying exact formatting rules, such as numeric-only values or handling of leading zeros.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific verb ('Decode') and a specific resource ('Cisco ISE / RADIUS failure or step code'), and gives concrete example codes. This clearly separates it from sibling tools that check, diagnose, or modify configurations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The intended use case—interpreting an ISE/RADIUS code—is implied by the description, but there is no explicit guidance on when to choose this over siblings like diagnose_dot1x or when not to use it. No alternatives or exclusions are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• latest_field_noteA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states that the tool reads a public RSS feed, implying a read-only external fetch rather than a network mutation. It does not detail failure modes, but for a simple feed retrieval this is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short and front-loaded with the core purpose in the first sentence. The second sentence adds useful sibling differentiation and source context, though the claim about touching the network is slightly ambiguous.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one optional parameter and an output schema, the description covers the purpose, content scope, and data source. It leaves out explicit guidance on the count parameter, but the output schema and simple schema reduce the need for more detail.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter, count, is never referenced in the description, and schema description coverage is 0%. The agent must infer from the parameter name and default value that count controls how many episodes are returned; the description adds no semantic value here.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the resource (This Week in NetOps episodes), the temporal scope (most recent), and the content domain (changes in networking, cloud, and automation). It also distinguishes itself from sibling tools by noting it is one of only two tools that touch the network along with check_tls_endpoint.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The intended use is implied: agents should call this when they need the latest weekly NetOps field notes. It names check_tls_endpoint as a sibling with network-touching behavior, but it does not explicitly state when to prefer this tool over that one or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explain_firewall_changeA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral burden. It reveals non-obvious behavior: changes are keyed by uuid to avoid confusing edits with delete-plus-add, save-stamp noise is recognized and set aside, and risk ratings include their stated grounds. It does not discuss permissions, limits, or side effects, but as a read-style explainer the disclosed detail is strong.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The main capability is front-loaded and every clause in the description carries high-signal information: uuid keying, plain-English rule explanations, risk grounds, shadowed-rule notes, API attribution, and save-stamp noise handling. There is no filler or repetition of schema field names.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex analysis tool with no annotations and no schema descriptions, this description covers the core domain, key behaviors, and important output semantics well. The main missing context is the 'device' parameter and explicit routing among sibling diff tools, so it falls just short of fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It indirectly defines 'before' and 'after' as OPNsense config.xml captures, but it never explains the optional 'device' parameter or how it relates to attributing 'which API call made the change'. With three parameters and zero property descriptions, this is a significant gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Explain a change between two OPNsense config.xml captures rule by rule'. It then enumerates distinct output facets such as added/removed/edited/moved rules, risk ratings, shadowed-rule notes, and API attribution, clearly differentiating it from generic siblings like explain_config_diff.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The intended context is implied by 'OPNsense config.xml' and the firewall-rule focus, but the description never explicitly states when to use this tool versus alternatives. It also does not name a sibling tool or provide any when-not-to-use guidance, so the agent must infer the boundary with generic config diff tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• extract_device_factsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral burden. It usefully discloses the accepted input formats and the extraction outputs, but it does not mention failure behavior, handling of unsupported formats, or whether the operation is purely local/read-only. The supported formats add some transparency, but richer caveats about parsing limitations would improve it.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tight sentences with no filler. The main action and outputs are front-loaded, and the config format list is compactly appended. Every sentence adds meaningful information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that an output schema exists and the input schema is simple, the description covers the key invocation context: what to pass and what to expect. It misses only guidance on the optional 'device' and 'vendor' parameters and edge cases, but these are minor gaps relative to the provided schema and output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for all three parameters. It clearly implies that 'config' is the device configuration text and hints at vendor formats, but it never explains the 'device' parameter and does not explicitly connect 'vendor' to parsing behavior. Two of three parameters remain effectively undocumented.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Pull') and resource ('device config'), then enumerates the exact facts extracted: hostname, inferred role, IPv4 addresses with subnets, subnets, and VLANs. It also names the supported config formats, which clearly distinguishes this parsing/extraction tool from siblings like explain_config_diff or check_config_compliance.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context by listing the supported device config formats (EdgeOS, MikroTik, OPNsense, EdgeSwitch/FastPath, Cisco IOS-style), making it apparent when this tool applies. It does not explicitly state when to avoid it or name alternatives, but the sibling tool names are distinct enough that an agent can infer the appropriate use case.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_tls_endpointA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden, and it discloses meaningfully: it reads the certificate the host 'actually serves' (live data, not config), accepts any port, and refuses private/reserved address resolutions — an important SSRF-style safety behavior. The enumerated findings also tell the agent what classes of issues it will surface. It omits failure behavior (timeouts, unreachable hosts, non-TLS ports), but the disclosed traits go well beyond the bare schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two dense sentences with zero filler: the first front-loads the action and finding set, the second adds the safety constraint. Every phrase earns its place, including the parenthetical finding list that would otherwise require a schema or docs lookup.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a network-dependent tool with an output schema, the core purpose, constraints, and safety behavior are well covered, and return values are handled by the schema. The gaps are the unexplained role parameter and the absence of failure-mode context (timeouts, connection refusal, non-TLS endpoints), which an agent would need to anticipate errors when invoking a live-network tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description is the only source of parameter meaning, and it does clarify host ('public host:port') and port ('Any port'). However, the role parameter is completely unexplained — its likely semantic (e.g., SNI name) is left to the agent to guess, which is a real gap given nothing else documents it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action sequence (connect to a public host:port, read the served certificate) and enumerates the exact findings (expiry, weak key, weak signature, self-signed, hostname not in SANs, CA expiry). It also distinguishes itself from the sibling find_certs_in_config by framing this as the live-endpoint variant, so an agent can tell them apart without opening the schema.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'run the same findings as find_certs_in_config on it' clearly situates this tool against its sibling, implying the config-checking alternative for offline analysis. The 'Refuses hosts that resolve to private or reserved addresses' constraint and 'Any port' scope give an agent concrete boundary conditions for when the tool applies. It stops short of an explicit when-not rule, but the routing signal is strong.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_certs_in_configA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden, and it does well: it discloses that the tool reads inline PEM blocks from multiple vendor formats and OPNsense config.xml, and enumerates the certificate health checks it performs. The read-only nature is evident from 'Find,' 'check,' and 'Reads,' though it does not discuss error handling or edge cases like malformed configs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is information-dense but not bloated: the first sentence states scope and checks, and the second adds supported formats and the differentiator about web monitors. Every clause contributes meaning, with no filler or repetition of schema details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool's parsing complexity and check variety are well covered, and the output schema presumably handles return-value structure. The only notable gap is the undocumented `role` parameter, though it is optional with a default, so the core invocation with just `config` remains clear enough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It provides useful context for `config` and `vendor` through examples like Cisco, EdgeOS, RouterOS, and OPNsense formats, but it never explains the `role` parameter at all, and it does not explicitly link the prose to the parameter names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource ('Find every certificate embedded in a device config and check it') and enumerates precise checks: expiry severity buckets, weak RSA keys, MD5/SHA-1 signatures, self-signed leaves, and expiring CAs. It also differentiates from sibling tools by noting these are certificates 'inside the config, not on a port,' which clearly separates it from check_tls_endpoint.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly establishes when to use the tool: when certificates are embedded in device configs or OPNsense config.xml blobs, and it explicitly contrasts with port-based certificates. However, it never names an alternative tool directly or gives an explicit when-not-to-use statement, so the routing is implied rather than stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• diagnose_dot1xA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It explicitly states 'Read-only, rule-based,' and explains what the tool will name and deliver, including fixes on both switch and ISE. It does not discuss failure behavior on unparseable input or limits on log size, but the core behavioral traits are disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and information-dense, front-loading the core purpose before enumerating inputs, root causes, and outputs. Every clause adds value; the final 'Read-only, rule-based' tag is a useful behavioral note without surplus wording.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a diagnostic tool with free-text log inputs, the description covers inputs, scope, root causes, and fix guidance well. An output schema exists, so return-value documentation is not the description's job. Minor gaps remain around handling malformed or empty input and when to prefer lookup_ise_failure_code for a single failure code, but the description is largely complete for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It maps the three inputs conceptually: RADIUS/ISE log, switchport config, supplicant log, and says 'any of' may be pasted. However, it does not give format hints, size expectations, or more explicit per-parameter guidance, leaving some ambiguity for an agent preparing the call.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Diagnose a port that will not authenticate,' then details the exact inputs (RADIUS/ISE log, switchport config, supplicant log) and outputs (root cause, fix). This clearly distinguishes it from broader diagnostic or parsing siblings like check_config_compliance or extract_device_facts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit context for when to use the tool: when a port will not authenticate. It also tells the agent what to paste. It does not explicitly name alternative tools or exclusion conditions, but the usage context is clear and actionable despite that.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• infer_topologyA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and does substantial work: it explains adjacency, subnet merging, the /16 cutoff, and the Mermaid output format. The main gap is that VLAN inference is mentioned up front but not explained in the same detail as subnet inference.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well structured: purpose first, then precise inference rules, each earning its place. There is no repetition of schema information and no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists, the description appropriately focuses on input and behavior. It covers the main algorithm rules and output type. The only substantive omission is the exact behavior for the VLAN portion of the stated purpose and any failure behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description complements the schema by clarifying that the single devices parameter is a set of configs whose interfaces are parsed for network adjacency. This adds semantic meaning beyond the schema's bare field definitions, though it does not mention the optional vendor property.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states exactly what the tool does: consume multiple device configs, infer subnet/VLAN adjacency, and produce a Mermaid diagram. It clearly distinguishes this from sibling tools like extract_device_facts or explain_config_diff by naming a specific output artifact and inference goal.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  "Given several device configs" provides a clear usage context, and the inference rules make it obvious when this tool applies. There is no explicit when-not-to-use guidance, but no sibling appears to perform topology inference, so exclusions are less critical.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• preflight_changeA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full behavioral burden and handles it well: it states 'Read-only; nothing is applied', which is the decisive safety trait, and it also discloses the conditional blast-radius behavior and the exact verdict vocabulary. No annotation contradiction exists.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two dense sentences front-load the operational context ('Before you push a change') and then enumerate the analysis steps, verdict, and safety property. Every clause earns its place; there is no filler or repetition of schema boilerplate.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a preflight gate with an output schema, the description is nearly complete: it names the verdicts, covers the core diff inputs, defines blast-radius conditions, and states read-only status. The remaining gaps are minor but real: device/vendor parameters are unexplained, and no sibling routing is given for diff-only or compliance-only needs.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaning for current/proposed (the diff pair) and fleet (configs of other managed devices for blast radius). However, schema description coverage is 0% and the description never mentions the 'device' or 'vendor' parameters, leaving those roles to be guessed despite the vendor options in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific verb and resource: diff a proposed config against current, risk-tag changes, identify compliance regressions, and compute blast radius, ending in a gate verdict. The 'gate verdict' framing clearly separates it from sibling tools like explain_config_diff and check_config_compliance, which only analyze or check rather than gate a change.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  'Before you push a change' gives an explicit trigger condition, and the 'if you pass the configs of the other devices you manage' clause defines when fleet input matters. It does not name alternatives or explicitly say when NOT to use this tool, so it falls one step short of the top score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explain_config_diffA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description takes on the full burden and does it well: it discloses determinism ('Deterministic — no model call'), privacy ('nothing leaves the machine'), preprocessing ('volatile lines stripped, save-stamp noise filtered'), and output behavior ('grouped by config section, each tagged with a risk level and why it matters'). This goes far beyond a simple 'explains diff' statement.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two dense sentences, front-loaded with the core action and resource. Every clause adds value: platform coverage, noise filtering, grouping, risk tagging, determinism, and data locality. No filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the essential behavioral context an agent needs: input domain, supported formats, processing quirks, output shape, and privacy/determinism guarantees. Since an output schema exists, the description does not need to detail the return structure, and the provided text is sufficiently complete for this moderately complex tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, so parameters are bare names. The description compensates by clarifying that 'before' and 'after' are full config snapshots and by listing vendor-specific formats that map to the 'vendor' parameter. It does not define 'device' explicitly, but the overall context is sufficient for correct invocation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource: 'Explain what changed between two network device config snapshots, in plain English.' It enumerates the supported platform formats and states the core output (grouped changes, risk levels), which clearly distinguishes it from siblings like explain_firewall_change or sanitize_config.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description makes the intended use clear: comparing two config snapshots and getting a plain-English explanation. It lists supported vendors/formats, giving an agent concrete criteria for when to select this tool, though it does not explicitly state when not to use it or name alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• sanitize_configA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure and does so richly. It lists exact redaction categories, explains deterministic placeholder mapping for IPv4/IPv6/MACs/hostnames/domains, notes preservation of subnet and wildcard masks, and states that it returns clean text plus per-category counts.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but well organized, front-loading the main purpose and then efficiently enumerating behaviors without filler. Every clause adds useful information, and the return behavior is stated at the end.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given a single string parameter, no annotations, and an existing output schema, the description provides enough context for an agent to select and invoke the tool correctly. It covers the input scope, transformation rules, and output summary without needing to duplicate schema-provided return details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There is only one parameter, config, and the schema provides no description, so the description must compensate for 0% schema coverage. It adds meaning by specifying what kind of content the config may contain and how it will be transformed, though it does not explicitly define the expected input format or encoding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Scrub a config before it goes anywhere' with detailed redaction categories. It clearly distinguishes this tool from network-focused siblings like check_config_compliance and explain_config_diff by defining a unique sanitization responsibility.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'before it goes anywhere' establishes the intended context: use this when a config must be shared or exported safely. It does not explicitly name alternatives or exclusions, but none of the listed siblings compete for the same purpose.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.