Server Quality Checklist

Tool Scores

• test_historyC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavior. It mentions output fields and service parameter options, but it omits read-only nature, ordering, pagination, error behavior, or side effects. This is insufficient for a tool that could be expected to have such behaviors.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very short and front-loads the core purpose, but it is partly a fragment ('Recent diagnostic runs (type, status, Passed/Failed, timestamps).') and mixes parameter semantics awkwardly in the second sentence. It is not verbose, but it lacks structured clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description must carry the full context. It gives a basic idea of return fields and one parameter but omits output format, limit semantics, and any guidance on how this fits with sibling diagnostic tools. The tool is not complex, but the description is still incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaning for the 'service' parameter (site alias | service_id | 'all'), but it does not explain the 'limit' parameter at all. Since schema description coverage is 0%, the description must compensate for both parameters, but it only partially addresses one.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the resource (recent diagnostic runs) and the fields returned (type, status, Passed/Failed, timestamps), which distinguishes it from sibling tools like test_result. However, it lacks an explicit verb such as 'list' or 'get', and it doesn't explicitly contrast with sibling tools, so it's not perfectly clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no guidance on when to use this tool versus alternatives. The description does not mention scenarios, prerequisites, or exclusions, and it doesn't reference any sibling tools. The agent is left to infer usage from the minimal context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_faultsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure, but it only states scope. It does not disclose return format, whether the tool is read-only, how include_details affects output, or any operational caveats. 'Open fault tickets' implies a read operation but is not explicit enough.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that is concise, front-loaded, and free of fluff. It communicates the core purpose and scope without wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one optional parameter and no output schema, the description is still incomplete: it does not explain the include_details parameter, what the response contains, or any prerequisites. The scope clarification helps, but there are clear gaps for an agent to invoke the tool confidently.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has one parameter, include_details, with 0% description coverage, and the description does not mention it at all. The name is somewhat self-explanatory, but the description provides no meaning beyond the schema, so it fails to compensate for the missing parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses 'Open fault tickets' to indicate listing open fault tickets, and adds a clear scope qualifier ('account-wide, not per-service') that differentiates it from siblings like service_outages. However, 'Open' is slightly ambiguous and could be read as an action rather than a listing operation, so it is not a perfect 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear scope ('account-wide, not per-service'), which implies when this tool is appropriate, but it does not explicitly state when to use it versus alternatives such as service_outages or list_diagnostics. There are no explicit exclusions beyond the per-service distinction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• speed_test_resultsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It does not disclose whether this is a read-only operation, what the response format is, or any limiting behavior. The only behavioral hint is the default 'all' for service.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short segments, front-loading the resource and then defining the parameter. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter query, the description provides the resource type and parameter semantics. However, it doesn't clarify the return format or how this relates to siblings like test_result, which would help context. The absence of an output schema increases the burden, but the simplicity makes this minimally viable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description explains the service parameter as accepting 'a site alias | service_id | ''all'' (default)', which conveys the allowed values and default. This adds meaning beyond the schema, which only has a type and default.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool as returning speed test results recorded against the service, with a mention of AussieBB context. However, it lacks an explicit verb (e.g., 'list' or 'get'), making it slightly less direct. It does distinguish from sibling tools by focusing on speed tests specifically, but does not name alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus sibling tools like test_result or test_history. The description only states what the resource is, leaving the agent to infer appropriate usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_diagnosticsB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavior. It indicates the listing includes which diagnostics are destructive, which is useful domain context. However, it does not explicitly state that the operation itself is read-only or describe the return format or error behavior. The description carries a moderate burden and partially meets it.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, using two short statements with no filler. The first part states the action and output fields, the second explains the parameter. It could be structured more clearly (the parameter note reads like an afterthought), but every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple listing tool with one optional parameter and no output schema, the description covers the main purpose and parameter. However, it does not explicitly mention the output format (e.g., an array of objects) or how to interpret 'destructive' in the results. Given the sibling tools, it could also hint at the workflow (list diagnostics then run one), but that is not strictly required.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema only defines 'service' with type string and default 'all'. The description adds valuable meaning: 'service: a site alias | id | "all"' explains the accepted values. This compensates for the 0% schema description coverage by providing essential usage guidance for the parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists diagnostics for a service, using the specific verb 'List' and the resource 'diagnostics AussieBB offers'. It distinguishes from siblings like list_services (lists sites) and run_diagnostic (executes a diagnostic). The parenthetical '(name, summary, which are destructive)' clarifies the output fields, though slightly ambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no explicit guidance on when to use this tool versus alternatives. It does not mention that it should be used before running a diagnostic, nor does it reference sibling tools like run_diagnostic or list_services. The only extra information is the parameter explanation for 'service', which is not usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• test_resultA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry behavioral disclosure. It implies a read-only fetch and specifies the source of test ids, but it doesn't describe error handling, authorization, or the return format. This is a moderate gap for a fetch operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded with the action. However, the second fragment 'service: a site alias | service_id' is cryptic and introduces confusion, so it's not as polished as ideal.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter fetch, the description covers the core usage and id provenance, but without an output schema it leaves return-value semantics and error behavior undocumented. This is adequate yet has clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must clarify both parameters. It explains that 'service' is a site alias, but the trailing 'service_id' is ambiguous and not a declared parameter. The test_id is only implied as 'by id', which is insufficient given no schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Fetch one test's current status/result by id', a specific verb+object that clearly distinguishes this from sibling test_history (listing tests) and run_diagnostic (launching tests). The parenthetical about the id source reinforces the tool's unique role.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides clear context by noting the id comes from test_history or a wait=False run, which tells the agent when this tool is applicable. It stops short of explicitly naming alternatives or exclusions, so it doesn't earn a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_diagnosticA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description takes on the burden of disclosing behavior. It does reveal that destructive tests (kick_connection, ntd_reset) require confirm=true, which is critical safety information. Yet it omits other behavioral aspects like return values, side effects, or the effect of the wait parameter, leaving the agent with an incomplete picture.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences with no filler. It front-loads the action, includes a warning, and a parameter hint, earning every word.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a generic escape-hatch tool, the description covers the core purpose, a key behavioral caveat, and a parameter hint. But it lacks explanation of wait/timeout semantics and return format, which are important for an agent to invoke it correctly. Given the absence of an output schema, the description should provide more operational context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description is the only source of parameter meaning. It explains service as 'a site alias | service_id' and gives examples for test, and indirectly clarifies confirm via the destructive test note. However, wait and timeout_s are left unexplained, so the description only partial compensates for the schema gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs any diagnostic by name, with specific examples, and positions it as an 'escape hatch' equivalent to the portal's Run Tests buttons. This distinguishes it from the specific sibling tools like ntd_status and loopback_test by framing it as the generic runner.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives contextual guidance: it is the generic run mechanism, and destructive tests require confirm=true. However, it does not explicitly name alternative tools or state when to prefer a specific sibling over this generic one, so it falls short of full when/when-not guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• service_outagesA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description conveys the tool's read-only lookup nature with a service selector. It does not disclose data freshness, access requirements, or the interpretation of resolved vs scheduled, but for a non-destructive query tool this is acceptable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences cleanly separate the core purpose from the parameter note, front-loading the actionable context. No unnecessary words or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter query tool with no output schema, the description covers what it does, when to use it, and the parameter format. It could mention output shape or sample values, but the simplicity limits the risk.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description compensates by explaining that 'service' accepts a site alias, service_id, or 'all' with a default. This adds real meaning beyond the bare string type, though concrete examples would be even better.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the resource (NBN/AussieBB outages) and the state categories (current, scheduled, resolved). It lacks an explicit verb like 'list' or 'get', but the intent is unambiguous and differs from local-gear diagnostic siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'check this before blaming local gear' explicitly tells the user when to consult this tool before local troubleshooting tools like check_connection or loopback_test. It does not name alternatives explicitly but sets clear priority context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• loopback_testA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It discloses duration (~2 min), safety (non-disruptive), and asynchronous behavior (wait=False returns immediately with a test id). This goes beyond a simple read/write hint and provides useful operational context, though it does not cover error cases or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the key action and purpose, then adds safety and asynchronous behavior. No wasted words; every clause earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a test tool with no output schema, the description covers the essential context: purpose, duration, safety, and async polling option. It could mention timeout behavior or what test_result returns, but those are arguably covered by the existence of sibling tools. The description is adequately complete for the moderate complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains 'service' as 'a site alias | service_id' and clarifies the effect of 'wait=False' (immediate return with test id). However, 'timeout_s' is not explained, leaving one of three parameters under-specified. Partial compensation, not full.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Run a Loopback test' that 'proves data passes both ways between AussieBB and the NBN box/node'. This specific verb+resource phrasing distinguishes it from sibling tools like check_connection or run_diagnostic, which likely have different scopes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use this tool (when you need to verify bidirectional data flow) and notes it is 'Safe, non-disruptive', but it does not explicitly state when to prefer this over alternatives like check_connection or run_diagnostic, nor does it provide exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_servicesA
  4.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It only says 'List services' and lists return fields, but does not state whether the operation is read-only, whether authentication is required, or if pagination/rate limits apply. This is a significant gap given the absence of annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the action ('List services'), and includes only essential information: the resource, the return fields, and the parameter behavior. Every sentence earns its place with no redundant details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, return fields, and the single parameter's meaning. However, it lacks notes on pagination, error handling, or authentication, which would be useful for a complete picture. Given the tool's simplicity, this is a minor gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema only defines service_type as a string with default 'NBN'. The description adds critical semantics by specifying the allowed values: 'NBN' (default) or 'all'. Since schema description coverage is 0%, this fully compensates for the schema's lack of detail.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists services on the AussieBB account and specifies the exact fields returned (service_id, site alias, plan, speed tier, address, WAN IPs). This distinguishes it from sibling diagnostic tools, which are focused on diagnostics and tests.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description establishes a clear context for use: whenever the agent needs to list account services. It does not explicitly compare to alternatives or state when not to use it, but the sibling tools are for diagnostics, making the use case unambiguous.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_connectionA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It explicitly states 'Safe, non-disruptive', which is equivalent to a read-only hint, and adds 'from AussieBB's end' to clarify the scope. It does not mention authentication or return format, but the key safety aspect is covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded, starting with 'Instant ISP-side connectivity check'. It includes safety, priority, and parameter guidance in a few short fragments. Slightly choppy but every part earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one optional parameter and no output schema, the description covers what it does, when to use it, and what the parameter accepts. The question 'is the service connected' implies a boolean answer, making return info less critical. Overall, it provides sufficient context for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema only defines 'service' as a string with default 'all'. The description adds crucial detail: 'a site alias | service_id | all (default)', fully explaining accepted values. This is high-value beyond the schema, especially given 0% schema description coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Instant ISP-side connectivity check' and 'is the service connected from AussieBB's end?', which specifies the verb (check) and resource (connection). It distinguishes itself from siblings like loopback_test and service_outages by emphasizing the ISP-side perspective and its role as the first diagnostic.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'THE first diagnostic to run', giving clear priority and usage context. It also notes the tool is safe and non-disruptive, implying it can be run without concern. It does not explicitly name alternatives or exclusions, but the context is strong.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ntd_resetA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden and excels: it flags 'DESTRUCTIVE', warns of 'minutes of outage', and discloses that it 'Refuses unless confirm=true'. It also notes the service parameter restriction ('never all'). This gives the agent a complete picture of the tool's impact and prerequisites.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is highly concise, front-loading the critical warning ('DESTRUCTIVE') and then covering usage, confirmation, and parameter constraints in three sentences. Every sentence adds distinct value with no repetition or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a destructive action with no output schema, the description covers the key aspects: what it does, when to use it, the outage impact, the confirmation requirement, and service restrictions. It does not describe return values or the behavior of wait/timeout_s, which would improve completeness, but the core context is sufficiently provided.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It explains two key parameters: service ('a site alias | service_id (never all)') and confirm ('Refuses unless confirm=true'). However, it omits wait and timeout_s, which are not self-evident given the destructive reboot context. It partially compensates but leaves gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states exactly what the tool does: 'reboot the NBN box' with a clear verb and resource. It also includes the scope ('minutes of outage') and a specific use condition ('Only useful when NTD Status errors'), which distinguishes it from diagnostic siblings like ntd_status or check_connection.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a clear condition for use: 'Only useful when NTD Status errors.' It also instructs to 'get the user's explicit approval first,' which is an actionable guideline. It does not explicitly name alternative tools, but the condition implicitly separates it from diagnostic tools, so it falls just short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ntd_statusA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and does well by stating the tool is 'Safe, non-disruptive', takes '~2 min', and explains the async behavior with wait=False returning a test id. It discloses the key behavioral traits that an agent needs, though it doesn't detail failure modes or other edge cases.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences that front-load the core purpose and expected duration, then followed by parameter and async behavior details. Every word adds value, with no redundancy or irrelevant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having no output schema, the description describes what the test returns (link state, uptime, flap count) and how to handle wait=False via test_result polling. This gives an agent a complete picture for a diagnostic tool, including safety and time expectations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explicitly explains the service parameter as 'a site alias | service_id' and clarifies the wait parameter's behavior with wait=False. The timeout_s parameter is not directly explained, but the mention of '~2 min' and schema defaults make it inferable. This is good but not complete for all parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs an 'NTD Status test' and specifies the specific outputs: 'NBN box link state, uptime, and dropout (‘flap’ count)'. This distinct purpose differentiates it from sibling diagnostic tools like check_connection and loopback_test.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description identifies a specific use context: 'the evidence AussieBB's fault process keys on' which indicates when this tool is relevant. It also provides usage guidance for wait=False polling via test_result. However, it does not explicitly mention when not to use it or name alternative tools, so it lacks explicit exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kick_connectionA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully discloses all critical behaviors: it is 'DESTRUCTIVE,' notes 'reconnection can take ~10 min,' and states the tool 'refuses unless confirm=true.' This goes beyond mere functionality to set accurate expectations for the agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three concise sentences, front-loaded with the DESTRUCTIVE warning. Every sentence carries essential information: purpose, use case, consequence, and parameter guidance. No waste or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a destructive tool with no output schema and no annotations, the description covers all necessary context: what it does, when to use it, how long reconnection takes, the confirmation requirement, and parameter constraints. This is complete for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description compensates fully. It explains 'service' as 'a site alias | service_id' and adds the crucial constraint 'never all.' It also clarifies the 'confirm' parameter by indicating it must be true for the action to proceed, which the schema only vaguely defaults to false.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action: 'drop the service's session at AussieBB.' It uses a specific verb ('drop') and resource ('service's session'), and the context (swapping modem/router) distinguishes it from sibling tools like check_connection or ntd_reset.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage is explicitly tied to a scenario: 'use when swapping the modem/router.' It also provides a necessary precondition: 'get the user's explicit approval first.' However, it does not name alternative tools or explicitly state when not to use it beyond the parameter constraint 'never all'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.