Server Quality Checklist

Tool Scores

• list_groupsB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the return format ('Formatted table') and what columns to expect, which is helpful. However, it doesn't mention important behavioral aspects like whether this requires authentication, has rate limits, returns all groups at once or uses pagination, or if there are any access restrictions. The description adds some value but leaves significant gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately concise with two clear sentences. The first sentence states the core functionality, and the second describes the return format. There's no wasted text, though the structure could be slightly improved by combining the two sentences more fluidly or adding a brief introductory phrase.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has zero parameters, 100% schema coverage, and an output schema exists, the description is reasonably complete for a simple read operation. However, with no annotations and a read operation that likely has behavioral considerations (authentication, data scope, etc.), the description should ideally mention at least basic context about access or limitations. The output schema will handle return structure details, but behavioral transparency remains a gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters (schema coverage 100%), so the description doesn't need to explain parameters. The baseline for zero parameters is 4, as there's no parameter documentation burden. The description appropriately focuses on what the tool does rather than parameter details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('List') and resource ('server groups'), including what information is returned (descriptions and member counts). It distinguishes from sibling 'list_servers' by focusing on groups rather than individual servers. However, it doesn't explicitly differentiate from other potential group-related operations that might exist in the future.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. While it implicitly suggests this is for viewing group information rather than executing operations (like 'execute_on_group'), there are no explicit when/when-not instructions or references to sibling tools. The agent must infer usage context from the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• upload_fileA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions SFTP (implying secure transfer) and the return format, but doesn't cover critical behavioral aspects like authentication requirements, error handling, file size limits, overwrite behavior, or network timeouts. For a mutation tool with zero annotation coverage, this leaves significant gaps in understanding how the tool behaves.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, args, returns) and uses minimal sentences that each serve a purpose. The first sentence states the core function, followed by organized parameter explanations and return information. It could be slightly more concise by integrating the parameter explanations more fluidly, but overall it's efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given a mutation tool with 3 parameters, no annotations, and an output schema (which handles return values), the description covers the basic purpose and parameters adequately. However, it lacks important contextual details like authentication requirements, error conditions, or performance characteristics that would be helpful for safe and effective use. The presence of an output schema reduces the need to explain returns, but other gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description fully compensates by explaining all three parameters with clear examples and context. It specifies 'server' as a server name with an example, 'local_path' as an absolute path to the source file, and 'remote_path' as an absolute destination path. This adds substantial meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Upload a file') and resource ('to a remote server via SFTP'), distinguishing it from sibling tools like download_file (which performs the inverse operation) and execute/execute_on_group (which run commands rather than transfer files). The verb+resource combination is precise and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context through the mention of SFTP and parameter explanations, but doesn't explicitly state when to use this tool versus alternatives like download_file or when not to use it (e.g., for local file operations). It provides basic guidance through parameter descriptions but lacks explicit comparative guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• download_fileA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the SFTP protocol and describes the return value, but doesn't cover important behavioral aspects like error conditions, timeout behavior, authentication requirements, file size limitations, or whether the operation is idempotent. The description provides basic operational context but lacks comprehensive behavioral transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose, Args, Returns) and uses minimal sentences. Each sentence earns its place by providing essential information. The formatting is efficient, though the 'Args:' and 'Returns:' labels could be slightly more concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (which handles return value documentation), no annotations, and 3 parameters with good description coverage, the description is reasonably complete. It covers the core operation, all parameters, and return format. The main gap is lack of behavioral context like error handling and authentication requirements, but the presence of output schema reduces the completeness burden.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description compensates by clearly explaining all three parameters with examples and context. The Args section provides meaningful semantics: 'server' is explained with an example format, 'remote_path' specifies it must be absolute, and 'local_path' clarifies it's the destination. This adds substantial value beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Download a file') and resource ('from a remote server via SFTP'), distinguishing it from sibling tools like upload_file. It provides a complete verb+resource+protocol combination that leaves no ambiguity about what the tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like upload_file or other sibling tools. It mentions SFTP protocol but doesn't specify prerequisites, authentication requirements, or when this tool is appropriate versus other file transfer methods. No exclusions or alternative scenarios are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_serversA
  4.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool returns a 'formatted table' which adds useful context about output format, but doesn't mention potential limitations like pagination, rate limits, or authentication requirements for accessing server configurations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly structured and concise: a clear purpose statement followed by well-organized Args and Returns sections. Every sentence earns its place by providing essential information without redundancy, and the information is front-loaded appropriately.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (1 parameter, read-only operation), the description is quite complete. It explains purpose, parameter usage, and output format. With an output schema present, it doesn't need to detail return values further. The only minor gap is lack of explicit mention that this is a read-only operation, though that's implied by 'List'.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description must compensate fully. It does this excellently by explaining the single parameter's purpose ('Optional group name to filter by'), behavior ('Shows all servers if omitted'), and relationship to other tools ('Use list_groups to see available group names'), adding substantial meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('List all configured SSH servers') and resources ('SSH servers'), and distinguishes it from siblings by mentioning 'groups and descriptions' which aren't covered by other tools like execute or download_file.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool vs alternatives: it mentions using 'list_groups to see available group names' for filtering, and specifies that the group parameter is optional for showing all servers, giving clear context for usage decisions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• execute_on_groupA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavioral traits: concurrency via max_parallel_hosts, per-server timeout behavior, fail_fast cancelation, force bypass of dangerous-command detection, dry_run preview, and return format. No contradiction with any missing annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear opening sentence, numbered args, and a return summary. It is slightly verbose in places (e.g., 'Each server has its own timer' could be shorter), but every sentence adds value. Overall, it's efficient and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (7 parameters, parallel execution, multiple flags), the description covers all necessary aspects: concurrency settings, per-server timeout, failure modes, dangerous-command detection, dry-run preview, and return format. The presence of an output schema reduces the need for detailed return description, and the provided summary is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must explain all parameters. It does so comprehensively: group (with reference to list_groups), command, timeout (with range and per-server independence), working_dir (with fallback to default_dir), fail_fast (with default behavior), force (with caution), and dry_run (with behavior description). Each adds meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the purpose: 'Execute a shell command on all servers in a group in parallel.' The verb 'Execute' and resource 'shell command on all servers' are specific. It distinguishes from siblings like 'execute' (likely single server) and 'list_groups' (discovery).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool (executing on a group) and references 'list_groups' for discovering groups. It lacks an explicit statement of when not to use it (e.g., for single server operations), but the sibling context implies that 'execute' would be appropriate then.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• executeA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully covers behavioral traits: dangerous-command detection (with force bypass), timeout range (1–3600 seconds), output truncation at max_output_bytes, working_dir fallback logic, and dry_run behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with an Args section and Returns, and every sentence adds value. It is somewhat lengthy but justified by the number of parameters and behaviors; minor trimming could improve conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 6 parameters, no annotations, and presence of output schema (mentioned in Returns), the description thoroughly covers all aspects: parameter details, defaults, edge cases, behavior, and return format. It leaves no critical gaps for the agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 0% description coverage, but the description adds detailed semantics for all 6 parameters: server format and validation, command as bash, timeout units and range, working_dir fallback, force bypass details, and dry_run preview purpose. It also explains return value formatting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Execute a shell command on a single SSH server.' It uses a specific verb ('execute') and resource ('shell command on a single SSH server'), and distinguishes from sibling tools like 'execute_on_group' which targets groups, and file operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly says 'on a single SSH server' and references 'list_servers' to see available servers, implying when to use this tool. It also notes 'dry_run' for validating intent before committing, and by context, alternatives like 'execute_on_group' exist for group execution.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.