Server Quality Checklist

Tool Scores

• collect_baseA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description should disclose behavioral traits. It only lists sub-commands run, but does not mention side effects (likely read-only), permissions, or failure modes. The 'cheap' hint suggests low overhead but is vague.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that front-loads the tool's purpose and lists included commands. No extraneous words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given a single parameter, comprehensive schema coverage, and sibling tools providing context, the description adequately covers the tool's purpose. Lacks mention of output format or side effects, but is sufficient for a simple 'collect' action.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The parameter 'host' is fully described in the schema with coverage 100%, and the description adds no extra detail such as format or constraints. Baseline score of 3 is appropriate as no further value is added.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool collects a group of specific commands (docker_ps, df, etc.) as a 'first look' at a host. It distinguishes from siblings like 'collect' and 'container_deep_dive' by specifying the scope and purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies it's for initial assessment ('cheap first look'), but does not explicitly state when to use versus alternatives like 'collect' or 'container_deep_dive'. No when-not-to-use guidance provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_hostsA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description conveys that the tool is read-only and optionally performs a TCP and SSH banner probe. This provides sufficient behavioral insight given the tool's simplicity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no unnecessary words. The first sentence covers purpose and optional behavior, the second explains output. Efficient and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description mentions the output is 'names to use with the other tools', but does not specify the exact format (e.g., list of strings). It also omits prerequisites (e.g., presence of hosts.json). For a simple tool, it is adequate but not fully detailed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and the schema already describes the parameter 'probe'. The description adds 'TCP + SSH banner' nuance but does not significantly expand beyond the schema, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists hosts from hosts.json and can optionally probe reachability. It mentions that it returns names for other tools, which distinguishes its role from siblings like collect or container_deep_dive, but does not explicitly differentiate.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by stating that the returned names are to be used with other tools, but it does not provide explicit guidance on when to use this tool versus alternatives or any exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• container_deep_diveA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries full burden and discloses that it runs docker_logs, docker_inspect (with redaction noted), container_state, and docker_stats. The explicit mention of '(redacted)' for docker_inspect adds transparency about data handling. However, it does not state whether the tool is read-only or has side effects (likely safe, but not stated).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence that is front-loaded with 'Shortcut:' and conveys the essential behavior efficiently. There is no wasted text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is adequate for a shortcut tool but lacks details on the return format/combined output since no output schema is provided. It does not explain what the agent should expect as a response or how the outputs from the subcommands are merged.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with clear descriptions for both required parameters ('host' from list_hosts, 'container' name). The description adds no extra meaning beyond what the schema provides, so baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it is a shortcut to run multiple Docker commands (docker_logs, docker_inspect, container_state, docker_stats) on a single container. It distinguishes from sibling tools like 'collect', 'collect_base', 'get_whitelist', and 'list_hosts' which serve different purposes (e.g., collecting data across hosts, listing hosts).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for one container as a shortcut, but does not specify when not to use it or provide explicit alternatives. No guidance on prerequisites or when to prefer individual commands over this composite tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• collectA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  In the absence of annotations, the description discloses key behaviors: commands are whitelisted (limited to up to 8), output is redacted and truncated, and invalid keys are rejected. No contradictions; it provides a clear safety profile for a read-like diagnostic tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences, front-loading the core action and constraints. Every sentence adds information without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations or output schema, the description covers core behavior and constraints but lacks details on output format or structure. For a tool with multiple parameters, more clarity on return values would improve completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 100% coverage with descriptions for all three parameters. The description adds value by linking keys to the whitelist and explaining the maxItems constraint (up to 8), which enhances schema meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The title 'Collect diagnostics by key' and description explicitly state it runs up to 8 whitelisted commands on a host and returns redacted/truncated output. It distinguishes itself from siblings like get_whitelist and collect_base by focusing on running commands vs listing keys or base collection.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions using get_whitelist to see valid keys and states that invalid keys are rejected, guiding proper use. However, it does not explicitly state when not to use this tool or provide alternatives for related tasks.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_whitelistA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses that it returns a complete set and lists the fields, indicating a read-only operation without side effects. Since no annotations are provided, the description adequately conveys the behavior, though it doesn't mention authentication or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, no redundancy, and immediately states the action and what is returned. It is well-structured and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of parameters and output schema, the description provides sufficient information: it lists the fields returned and emphasizes completeness. This fully meets the needs for a simple listing tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has no parameters, but the description adds value by detailing the return fields (key, purpose, shell template, container argument flag), which is not specified in the schema or output schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states it returns every command the server can run with specific details (key, purpose, shell template, container argument requirement). It distinguishes from sibling tools which involve collecting or listing hosts, clearly indicating this is for listing the command whitelist.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description makes it clear that this tool is for retrieving the complete set of executable commands. While it doesn't explicitly state when not to use it or provide alternatives, the context and sibling tool names imply its standalone role for whitelist inspection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.