Server Quality Checklist

Tool Scores

• list_deploymentsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It only says 'list recent workflow runs' without stating that it is safe/read-only, whether it affects state, or any other side-effect or access requirements. This is minimal transparency beyond the core action.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that is direct and to the point. No filler or redundant detail, making it highly efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool, the description is adequate but lacks detail about return format, pagination, or the time window implied by 'recent'. The name mismatch with 'deployments' also creates ambiguity. Given the absence of an output schema and annotations, slightly more context would be needed for full completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with all three parameters (repo, limit, workflow) described. The tool description adds no extra param semantics beyond implying recency ('recent') and workflow filtering, but the schema already covers these. Thus baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear action ('List') and resource ('recent workflow runs') with context ('for a repository'). However, the tool name 'list_deployments' clashes with the description's 'workflow runs', which could confuse agents expecting deployment objects. It is specific enough to be understood, but the naming inconsistency prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given on when to use this tool over siblings like get_deployment or deploy. The description simply states what it does without specifying scenarios, prerequisites, or exclusions, leaving the agent to infer usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_deploymentB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It only states 'Get one workflow run by id' without mentioning error handling, return format, auth requirements, or any other behavioral traits. This is insufficient for a tool with no structured safety hints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with zero extraneous words. It efficiently states the action and identifier, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations, lack of output schema, and a potential naming mismatch, the description is too sparse to be fully self-contained. It does not explain what the returned object looks like, error cases, or how this tool fits into the broader workflow deployment context, leaving the agent with limited guidance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already fully describes both parameters (repo and run_id) with 100% coverage, so the description adds no additional meaning. Per guidelines, baseline for high schema coverage is 3, and the description merely says 'by id' without elaborating on parameter formats or relationships.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Get one workflow run by id'), clearly distinguishing from listing siblings by focusing on a single item. However, there is a slight naming mismatch between the tool name ('get_deployment') and the described resource ('workflow run'), which could cause confusion but doesn't obscure the core purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you need a single workflow run by ID, but it does not explicitly contrast with alternatives like list_deployments or resolve_ref, nor provide context on when not to use this tool. The guidance is minimal and relies on inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• deployA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must carry the full burden of behavioral disclosure. It reveals a critical behavior: the pipeline verifies the receipt before releasing anything, which is a gating mechanism. However, it does not disclose other important traits such as whether the deployment is asynchronous, what side effects occur, or what happens on failure. The single behavioral detail is useful but incomplete.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: two sentences that front-load the primary action and follow up with the critical receipt requirement. Every word earns its place, and there is no redundant or vague language.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is a deployment operation with six parameters and no output schema, yet the description does not explain what the tool returns or how the receipt is obtained. While the schema documents all parameters well, the missing information about return values and the receipt issuance workflow leaves gaps. For a complex operation like deployment, a score of 3 reflects the incomplete context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides 100% coverage with detailed descriptions for all parameters, including the note that the built commit is `sha` not `branch`, and that `receipt_id` is injected by the gateway. The description itself adds no additional parameter semantics beyond the schema. With full schema coverage, a baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: deploying a specific commit through a deployment pipeline. It uses a specific verb and resource, and the mention of the receipt requirement adds distinguishing detail. It is unambiguously different from the sibling tools, which are about resolving refs, listing deployments, getting deployments, and listing environments.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states a key prerequisite: a receipt is required and verified before any release. This gives clear usage context. However, it does not explicitly mention when not to use this tool or name alternatives, such as using list_deployments to check status instead, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_environmentsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses a behavioral trait: environment names follow the host's own vocabulary, not a fixed set. As a listing operation, read-only is implied, but no annotations are available to confirm safety or other behaviors.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences: the first states the primary action and target, the second adds an important qualifier. No redundant details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a straightforward single-parameter list operation, the description covers the core purpose and a key caveat. However, with no output schema, it doesn't explicitly describe the return shape, though this is reasonably inferable from 'list environments'.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema fully documents the single required parameter 'repo' with type and description. The description adds no additional parameter semantics beyond the schema, so baseline applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the action ('List') and the resource ('deployment environments a repository defines'), distinguishing it from siblings like list_deployments by focusing on environments defined by the repo. The added nuance about host-specific vocabulary further clarifies the scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides context that environment names are host-specific, implying the user should not assume fixed values. However, it does not explicitly state when to prefer this over list_deployments or get_deployment, though the 'environments' vs 'deployments' distinction offers implicit guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• resolve_refA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of disclosing tool behavior. It adds useful context about deploy authorization but does not explicitly mention read-only nature, failure modes, or permissions, leaving gaps in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, with the purpose front-loaded and a critical usage directive appended. Every word earns its place; no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 2-parameter tool with no output schema, the description conveys purpose, usage context, and the result type (concrete commit SHA). It lacks error behavior details or explicit read-only confirmation, but those are reasonable omissions for such a simple tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already describes both parameters with 100% coverage. The description does not add new semantic details about parameters beyond the schema, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Resolve') with a clear resource ('a branch, tag or commit to a concrete commit SHA'). This directly conveys the tool's function and distinguishes it from sibling tools like list_deployments or deploy.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs 'Call this BEFORE deploying' and explains the reasoning (deploy is authorised for one specific commit). This provides a clear when-to-use directive, even though no alternative tools are named.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.