Server Quality Checklist

Tool Scores

• list_providersA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided. Description only mentions output format (YAML string) but reveals no side effects, authentication needs, or other behavioral traits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, zero waste. Front-loaded with action and outcome. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter list tool with an output schema, the description is mostly complete. Could be more specific about 'provider information' but overall adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist, so schema coverage is 100%. Baseline for 0 params is 4. Description adds nothing beyond schema, which is sufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List all configured cloud providers and their accounts/directories,' using a specific verb and resource. It distinguishes from sibling tools which focus on VMs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives. The context implies it is for listing providers, distinct from VM operations, but lacks direct usage instructions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• reboot_vmA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It states the action but does not disclose side effects (e.g., downtime), required permissions, or behavior if VM is already stopped.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences plus structured args/returns, no fluff, front-loaded with the core action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Adequately covers param format and return, but lacks preconditions (e.g., VM must be running), error handling, and permission context, which are important for a disruptive action.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so description compensates by specifying the composite VM ID format: 'provider:tenant:region:instance_id', which adds critical context beyond the schema's bare type field.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action 'Reboot a virtual machine' and distinguishes it from sibling tools like start_vm and stop_vm by focusing on restarting an already running VM.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for restarting a VM but provides no explicit guidance on when to choose reboot over start/stop, nor when not to use it (e.g., when VM is off).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• stop_vmA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It mentions the force option and its effect (force stop without graceful shutdown) but does not disclose potential data loss, permission requirements, or whether the operation is reversible. Some behavioral context is present, but gaps remain.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, with only three sentences for the main description plus an Args/Returns section. It is front-loaded with the core purpose. However, the structure could be slightly improved with clearer separation of lines.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (two parameters, no enums, minimal output), the description covers the key points: purpose, parameter details, and return format. It could mention prerequisites (e.g., VM must exist) or side effects, but overall it is sufficient for a basic operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description adds meaning beyond the schema by specifying the format for vm_id (provider:tenant:region:instance_id) and explaining the force parameter's default and effect. This helps an agent understand parameter usage beyond type constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Stop a virtual machine,' which is a specific verb-resource pair. It distinguishes the tool from sibling tools like start_vm and reboot_vm, as stopping is a distinct operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool (to stop a VM) but does not provide explicit guidance on when to use force vs. graceful stop, nor does it differentiate from alternatives like reboot_vm or start_vm. Missing context for usage scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_vmsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description carries full burden. It states the tool lists VMs and returns a YAML string, but lacks details on behavior when no filters are given (implies all VMs), performance, pagination, or permissions. Adequate but minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with a clear header and structured Args/Returns sections. Every sentence adds value with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no required parameters and an existing output schema, the description covers the tool's purpose, filters, and return format. Missing details like default behavior (list all VMs) are minor but don't hinder understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% (no parameter descriptions in schema). The description adds meaning for all three parameters: provider filters by 'aws' or 'azure', tenant by alias, region with examples. This compensates well, though it could specify case sensitivity or invalid values.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List all virtual machines across configured cloud providers,' which is a specific verb and resource. It differentiates from siblings like get_vm_details (details of a specific VM) and list_providers (list providers), making its purpose distinct.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions optional filters but does not explicitly state when to use this tool versus alternatives (e.g., when to use get_vm_details instead). The context of sibling tools provides implicit guidance, but no direct comparison or when-not-to-use advice is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• start_vmA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavior. It states it starts a VM and returns a YAML success/failure string. However, it lacks details like prerequisites (VM must be stopped), permissions, or side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with no extraneous words. The purpose is front-loaded, and the format follows a clear Args/Returns structure.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple one-parameter tool and presence of an output schema, the description covers the essential input format and return type. It could benefit from mentioning any preconditions (e.g., VM must be in stopped state), but it is largely adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage for the single parameter. The description adds value by specifying the required format 'provider:tenant:region:instance_id', which is critical for correct invocation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Start a virtual machine' – a specific verb and resource. It distinguishes from siblings like stop_vm and reboot_vm, providing unique purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not explicitly state when to use this tool versus alternatives (e.g., reboot_vm or stop_vm). The purpose is implied by the name and context, but no direct guidance is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_vm_detailsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full responsibility. It states the return format (YAML string containing details or error) and specifies the vm_id format, implying a read-only, idempotent operation. It could be improved by explicitly noting no side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two short sections. The first sentence immediately states the purpose, followed by parameter and return details. No unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With one parameter and an output schema (though not shown), the description covers parameter usage and return type. It lacks error condition details, but for a simple get tool, this is adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, but the description adds significant meaning by explaining the vm_id composite format ('provider:tenant:region:instance_id'). This helps an agent correctly construct the parameter beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the function: 'Get detailed information about a specific virtual machine.' The verb 'get' and noun 'details' precisely indicate a read operation, distinguishing it from sibling tools like list_vms (list all) or start_vm (mutate).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description specifies that the tool retrieves details for one VM. While it doesn't explicitly state when not to use it, the sibling context (list_providers, list_vms, etc.) provides implicit guidance. A brief mention of alternatives would improve clarity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.