Server Quality Checklist

Tool Scores

• analyze_failuresB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations are negative (readOnlyHint false, destructiveHint false), so they do not fully characterize side effects. The description adds some behavioral color by specifying deterministic output and optional host-model hypotheses, but it does not disclose persistence, mutation, or invocation side effects beyond that.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single front-loaded sentence with a clear verb and no filler words. It is concise and readable, though the jargon ('deterministic issues', 'host hypotheses') is dense and would benefit from a short expansion.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema, 100% parameter description coverage, and simple Boolean/string parameters, the definition is minimally sufficient for invoking the tool. However, it lacks usage context, prerequisites beyond schema hints, and behavioral detail, so an agent may not know precisely when to call it or what side effects to expect.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so both diagnosis_id and use_sampling are already documented in the schema; the tool description adds no parameter-level meaning. Baseline of 3 applies because the schema carries the full burden.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific action ('Build') and concrete outputs ('deterministic issues' and 'optional host hypotheses') scoped to 'known evidence IDs', so the tool's purpose is clear and distinct from sibling tools like list_diagnoses or execute_probe. It stops short of a 5 because 'known evidence IDs' is undefined and no sibling is explicitly contrasted.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage is only implied: the schema notes diagnosis_id comes from start_diagnosis, which suggests ordering, but the description never says when to choose analyze_failures over get_diagnosis, discover_profile, or build_runbook_report. There are no exclusions, conditions, or alternative tool mentions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• build_runbook_reportB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The annotations already indicate the operation is neither read-only nor destructive. The description adds that the output is redacted and is written to the external state directory, but doesn't disclose whether existing runbooks are overwritten or what other side effects occur.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no wasted words. It front-loads the action, output type, and destination, and every word contributes to the tool's meaning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool with an output schema, the description plus schema provides enough to invoke it correctly: the required input is diagnosis_id from start_diagnosis, and the behavior is writing a redacted report to the external state directory. It doesn't spell out the full workflow, but the dependency is captured by the parameter description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%: the single parameter diagnosis_id is documented as an identifier returned by start_diagnosis. The description adds no further parameter meaning, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the action ('Write'), the deliverable ('redacted Markdown and JSON runbook'), and the destination ('external state directory'). This is specific enough to distinguish it from the sibling diagnosis/analysis tools, though it doesn't explicitly name which sibling it is not.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No usage guidance is provided in the description. It doesn't say when to call this tool versus siblings like get_diagnosis or analyze_failures; the only hint is the schema note that diagnosis_id comes from start_diagnosis, which implies a post-diagnosis workflow but is not stated as explicit guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_diagnosisB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false, so the safety profile is covered. The description adds a useful high-level view of the returned content, but it does not disclose additional behavioral traits such as auth requirements, error behavior, or side effects. It also does not contradict the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One concise sentence with no filler or repetition. The verb is front-loaded and the four return components are listed compactly, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With a single well-documented parameter, an output schema, and annotations covering the read-only safety profile, the definition is largely sufficient for a simple fetch-by-ID operation. The enumeration of components communicates the aggregate nature of the diagnosis, though adding workflow context about when to retrieve a completed diagnosis would make it even more complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the only parameter already documented as an identifier returned by start_diagnosis. The tool description itself adds no further parameter-level meaning, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Return') and names the resource as an aggregate of four concrete components: profile, preflight checks, probe evidence, and cited issues. This makes the tool's function fairly clear, though it does not explicitly differentiate it from sibling tools like list_diagnoses or start_diagnosis.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no explicit guidance on when to use this tool versus alternatives. The only indirect hint is in the parameter description ('Identifier returned by start_diagnosis'), which implies a workflow, but the main description does not state when get_diagnosis is the right choice compared to list_diagnoses or the other sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• discover_profileA
  3.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations provide readOnlyHint=false and destructiveHint=false, and the description adds that the tool refreshes state and invalidates stale diagnostic results. This clarifies the mutation side effect and does not contradict the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no filler. It is appropriately sized for a one-parameter tool, though it is somewhat terse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema and full parameter coverage, the description doesn't need to explain return values or parameters. However, it omits prerequisites, when refreshing is valid, and how invalidation affects the diagnosis workflow.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%; diagnosis_id is already documented as 'Identifier returned by start_diagnosis.' The description adds no additional parameter detail, so the baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description identifies a specific action ('Refresh'), a target resource ('dynamic dependency profile'), and a consequence ('invalidate stale diagnostic results'). It distinguishes this from siblings like start_diagnosis or get_diagnosis, though it doesn't define what the profile is.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'invalidate stale diagnostic results' implies usage when prior diagnostics may be stale, and 'Refresh' suggests a maintenance/update action. However, it does not explicitly name alternatives or state when not to use this tool, leaving the agent to infer the trigger.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• start_diagnosisA
  3.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds meaningful behavioral context beyond the annotations: it tells the agent the session is 'bounded' and 'secret-redacted'. Since annotations only indicate readOnlyHint=false and destructiveHint=false, these descriptors help clarify the operational nature of the tool. It could disclose side effects more fully, but the core behavior is adequately conveyed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence with no filler or redundant restatement of schema details. Every word earns its place and the most important behavioral descriptors are front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 100% schema coverage and an output schema present, the description plus schema provide enough information to invoke the tool correctly. The main missing element is explicit lifecycle guidance relative to sibling tools, but that gap is already captured under usage_guidelines.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so all three parameters are already fully documented in the schema. The description adds no additional parameter-level meaning, but given the complete schema coverage, no further compensation is required.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear action ('Create') and a specific resource ('bounded, secret-redacted local development diagnosis session'), which conveys what the tool does and distinguishes it from the sibling read/analysis tools. It is not tautological, though 'diagnosis session' could be slightly more explicit for an agent unfamiliar with the domain.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives no guidance on when to use this tool versus siblings like run_preflight, prepare_probe, or execute_probe. The agent must infer from the name and context that this is the initial step of a diagnosis lifecycle, but no explicit routing is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• prepare_probeA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With readOnlyHint=false and destructiveHint=false, the annotations already signal a non-read-only but non-destructive mutation. The description adds useful behavioral context by specifying the token is one-time and the plan is exact, which are not present in the schema or annotations. It does not mention token expiration or authorization requirements, but the annotation coverage lowers the burden.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single front-loaded sentence contains no filler. Every word contributes: 'exact' scopes the plan, 'discovered-command' ties to discovery, and 'one-time approval token' captures the security-relevant output.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The output schema exists, so return values do not need explanation, and the input schema covers all four parameters. The main gap is workflow order: the description never tells the agent that prepare_probe should be called after discover_profile and before execute_probe, which is important for selecting among the eight sibling tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so each parameter already carries a meaningful description and baseline is 3. The tool description does not repeat or add parameter-level meaning beyond calling the artifact a 'discovered-command plan', which loosely ties to candidate_id without explaining formats or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Create') and a clear resource ('an exact discovered-command plan and a one-time approval token'), which clearly distinguishes this from sibling actions like execute_probe, discover_profile, or run_preflight. An agent can tell this is the preparation step without opening schemas.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'discovered-command plan' and 'approval token' implies this sits after discovery and before execution, so the usage context is inferable. However, it never explicitly says 'use before execute_probe' or names conditions where an alternative should be chosen.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• execute_probeA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already mark the tool as not read-only and not destructive; the description adds that it persists evidence, limits output ('bounded'), and redacts it, and that exactly one probe is executed. This gives the agent a meaningful safety/effect profile beyond the structured hints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single front-loaded sentence contains the action, scope, and output behavior with no filler. Every word contributes to the decision to invoke the tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 5-required-parameter workflow tool, the description plus fully documented schema and output schema cover most invocation needs. It could explicitly state the prerequisite flow from prepare_probe, but the parameter descriptions already name prepare_probe as the source of plan_id and approval_token.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All five parameters are fully described in the schema (100% coverage), so the description does not need to define each one. It only loosely maps 'approved' to the approval parameters and says nothing about diagnosis_id or plan_id beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb-object pair ('execute one approved argv probe') and adds a concrete outcome ('persist bounded, redacted output evidence'). This clearly distinguishes execution from prepare_probe and other diagnosis/probe siblings without requiring schema inspection.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The word 'approved' and the presence of approval_token/plan_id in the schema imply this is the post-approval execution step, but the description never states 'use after prepare_probe' or contrasts it with run_preflight/analyze_failures. Usage is inferred rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_preflightA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description meaningfully discloses that the tool does not execute commands, which is a significant behavioral trait beyond the annotations. It also names the specific state categories being checked. It does not fully clarify whether any state is modified despite readOnlyHint=false, but it does not contradict the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, compact sentence that front-loads the action and resource, then adds a key differentiator. Every word earns its place with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With one required parameter, full schema coverage, and an output schema present, the description covers the essential behavior and scope. It is slightly thin on usage context, but the tool's simplicity and structured metadata prevent any critical gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%: the sole parameter, diagnosis_id, is already described as 'Identifier returned by start_diagnosis.' The description adds no additional parameter-level detail, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Check') and specific resources ('executable, environment-name, and loopback port state'), making the tool's purpose immediately clear. It also distinguishes itself from command-executing siblings by adding 'without command execution'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this is a preflight verification step before command execution, and the required diagnosis_id ties it to start_diagnosis. However, it does not explicitly state when to use it versus alternatives like prepare_probe or execute_probe, nor does it state when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_diagnosesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false, so the description does not need to restate safety. It adds valuable behavioral context by noting that projects are not read, implying a cheaper or more targeted operation. This goes beyond the annotations without contradicting them.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no redundant words. Every part earns its place: the action, the object, and the distinguishing non-behavior.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a parameterless listing tool with an output schema and clear read-only annotations, the description is fully sufficient. Nothing an agent needs to select or invoke this tool correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so there is no parameter burden for the description to carry. The schema coverage is complete and vacuous, and with no parameters the baseline of 4 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb ('List') and a specific resource ('persisted diagnosis sessions'). It also adds a distinguishing qualifier ('without reading their projects') that differentiates it from sibling tools like get_diagnosis, which might imply deeper project access.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'without reading their projects' provides useful context that this is a lightweight listing operation, but it does not explicitly name alternatives or state when a sibling should be used instead. The context is clear, though the explicit 'when not to use' guidance is absent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.