Server Quality Checklist

Tool Scores

• fleet_healthB
  3.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already mark the tool as read-only, idempotent, and non-destructive. The description adds substantial behavioral semantics beyond that: severity is policy metadata not impact or confidence, INDETERMINATE has a bounded rejection category, rejection severity HIGH means no trustworthy verdict, NO_DATA must be distinguished from INDETERMINATE, and advice is approval-required and non-executable. This effectively discloses how to interpret outputs and what not to claim.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a long, dense paragraph of caveats with no structural organization. While each sentence carries warning value, the accumulation of overlapping disclaimers (severity, rejection, no inference) obscures the core function and makes it hard to scan. The key purpose is buried under behavioral restrictions.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description provides rich interpretative context for outputs: point evaluation, severity semantics, INDETERMINATE categories, NO_DATA distinction, and non-executable advice. However, it fails to state what the tool actually returns and does not document the 'at' parameter's semantics. Given the output schema exists, the description covers many caveats but leaves a meaningful gap for a single-parameter tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter 'at' has no schema description, and the description does not explicitly explain its format or meaning. The phrase 'Point tools return a point evaluation' weakly implies the parameter selects a point in time, but it never confirms that 'at' is a timestamp or describes accepted formats. With 0% schema coverage, the description should compensate more directly; it provides only marginal added meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description does not clearly state a specific verb and resource (e.g., 'Get fleet health at a point in time'). It opens with 'Trusted-local, non-production, non-live read-only simulation' and mentions 'Point tools return a point evaluation,' which implies the purpose but relies heavily on the tool name. The contrast with investigate_timeline helps, but the primary function remains vague.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description distinguishes this tool from investigate_timeline by noting the half-open [frm, to) window versus point evaluation, and warns against inferring transitions or root causes. However, it never explicitly states when to choose this tool over the siblings or provides clear exclusion criteria. Usage guidance is implied rather than direct.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• triage_serverA
  3.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description goes far beyond the readOnly/idempotent annotations by explaining interpretation pitfalls: severity is policy metadata not impact, HIGH rejection means no trustworthy verdict, INDETERMINATE has a bounded rejection category, and advice is non-executable. These are critical behavioral disclosures that prevent misuse, fully leveraging the description's responsibility.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single dense paragraph with many caveats. Each sentence adds important interpretation context, but it lacks a clear front-loaded purpose statement and could benefit from bullet points or segmentation. It is appropriately sized for the complexity but not optimally structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers numerous critical caveats about output interpretation (severity, rejection, NO_DATA vs INDETERMINATE, root cause) that are essential for correct use. However, it doesn't explicitly state how the tool behaves for a given hostname or the exact return format, though the output schema likely covers that. Overall, it is quite complete for the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not explain the 'at' parameter (presumably a timestamp) or how hostname is used. While hostname is self-explanatory, the lack of any parameter explanation leaves the 'at' parameter ambiguous, and the description does not compensate for the low schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly indicates this is a point evaluation tool for triage, contrasting with investigate_timeline's half-open window. It identifies the tool as a read-only simulation and mentions returning a point evaluation, which gives a clear sense of the action and resource, though not in a single verb+resource phrase.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It contrasts this tool with investigate_timeline by noting investigate_timeline uses a half-open [frm, to) window, implying this tool is for point-in-time evaluation. However, it doesn't explicitly state when to use this tool versus alternatives, nor does it mention fleet_health. The numerous 'Do not' statements are usage constraints but not alternative selection guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• investigate_timelineA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the readOnly/idempotent hints, the description discloses that this is a 'trusted-local, non-production, non-live' simulation, explains severity is 'categorical policy metadata' not impact or confidence, and details INDETERMINATE and rejection semantics. No contradiction with annotations; it enriches them.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but every sentence serves a purpose, covering scope, semantics, and pitfalls. It is not redundant or bloated, and it leads with the core nature of the tool. The structure supports skimming for critical constraints.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity and existing output schema, the description covers the necessary behavioral context: temporal window semantics, severity interpretation, INDETERMINATE categories, NO_DATA vs INDETERMINATE, and reservation/advice limitations. It is sufficiently thorough for safe and correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema coverage, the description compensates by clarifying frm/to as a half-open window, which is essential for correct use. Hostname is not explicitly discussed, but its role is implied by the tool name and its type is in the schema. The description adds meaningful semantics to the key temporal parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool is a 'read-only simulation' for investigating timelines, using a half-open [frm, to) window. It clearly distinguishes itself from 'point tools' (siblings) by the temporal evaluation scope. This is a specific verb+resource with clear differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly contrasts point tools with the windowed behavior of investigate_timeline, indicating when the tool is appropriate. The numerous 'do not' instructions (e.g., 'do not infer exact transition or root cause', 'do not make a reservation claim') provide critical usage guardrails.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.