Server Quality Checklist

Tool Scores

• destroy_deploymentB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must fully disclose behavior but only states the action. It omits critical details such as whether the deletion is irreversible, what happens to associated resources, and required permissions. For a destructive tool, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with two sentences, no redundant information, and front-loads the core action. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool is destructive and has no annotations, the description is too sparse. It lacks essential context such as irreversibility, impact on related resources, and failure conditions. The existence of an output schema does not compensate for missing safety information.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaning to the single parameter by stating its role ('the name of the deployment to destroy'), which goes beyond the schema's bare name. However, it does not clarify format, validation rules, or error conditions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb (tear down, delete) and the resource (deployment), with the key identifier (by name). It effectively distinguishes from sibling tools like provision_task (create) and get_deployment_status (check status).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives, no prerequisites, and no warnings about irreversible effects. The agent is left to infer usage context without explicit direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• provision_taskA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description implies a write/mutation operation but provides no details on side effects, permissions, idempotency, or failure behaviors. With no annotations, this is a significant gap in behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is highly concise: a single sentence stating the purpose, followed by a bullet list of arguments. No wasted words, and the key action is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having an output schema (not shown), the description does not mention return values or usage context (e.g., what happens on success/failure, prerequisites). For a provisioning tool, more context about the result is expected.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds brief parameter explanations (e.g., 'A unique name for this deployment instance'), but these add only minimal meaning beyond the param names. Since schema description coverage is 0%, some compensation is needed, but the descriptions are not particularly insightful.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action: 'Provision a new deployment for a given task.' It uses a specific verb ('provision') and resource ('deployment for a task'), distinguishing it from siblings like list_tasks or destroy_deployment.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not explicitly state when to use this tool or when to avoid it. It provides parameter explanations but no guidance on when to choose this over alternatives like destroy_deployment.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_deployment_statusA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided. Description implies a read-only operation but does not disclose error handling (e.g., if deployment not found) or permissions needed. Basic transparency is present.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Extremely concise: one sentence plus an Args line. Purpose is front-loaded. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity, the description covers the basics but lacks integration hints (e.g., check status before destroy) and error scenarios. Output schema is external but not referenced.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Single parameter 'deployment_name' has 0% schema coverage, but description adds meaning: 'the unique name of the deployment to inspect'. This compensates well for the trivial parameter type.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states 'Get the current status, configurations, and outputs of a deployment by its name', specifying both the verb (Get) and the resource (deployment), distinguishing it from siblings like list_tasks and destroy_deployment.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool vs alternatives (e.g., before destroy_deployment) or prerequisites. Only states 'by its name' without contextual cues.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_task_schemaA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It correctly indicates a read operation ('Get'), but does not disclose potential errors (e.g., task not found) or any side effects. Adequate but lacks depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: two sentences with no wasted words. The first sentence is a clear one-liner defining the purpose, and the second describes the single parameter. Front-loaded and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, output schema available), the description is mostly complete. It covers the basic behavior, but could mention that the task must exist or the tool will return an error. Still adequate overall.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds 'The unique alphanumeric name of the task.' which clarifies the expected format and uniqueness of the task_name parameter, providing meaningful context beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Get the metadata and input schema (JSON Schema) for a registered task.' It uses a specific verb and resource, and naturally distinguishes from sibling tools like list_tasks (which lists tasks) and provision_task (which creates tasks).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for retrieving schema but provides no explicit guidance on when to use this tool versus alternatives (e.g., before provisioning). No exclusions or prerequisites are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_tasksA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full burden. It discloses that the tool returns a JSON string and supports optional filtering, which implies a read-only operation. However, it does not explicitly confirm idempotency or safety, though 'list' implies no side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with three clear sections: purpose, Args, Returns. Every sentence adds value, and the structure is front-loaded with the primary action. No unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given an output schema exists (though not detailed here), the description adequately covers the return type ('JSON string') and parameter purposes. It does not mention error handling or empty results, but for a list operation with simple filtering, this is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% per context, but the description adds meaning to both parameters: it explains they are optional filters and provides examples (e.g., 'database, network, compute' for category; 'aws, azure, gcp' for provider). This goes beyond the schema's type and title alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List all available Terraform tasks (templates) in the registry.' The verb 'list' and resource 'Terraform tasks' are specific, and the scope 'registry' is provided. It distinguishes from siblings like provision_task (deploy) and get_task_schema (schema retrieval).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description indicates optional filters (category, provider) but does not explicitly state when to use this tool relative to alternatives. It lacks guidance on prerequisites or situations where listing is appropriate. The sibling names imply differentiation, but the description itself offers no usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.