Server Quality Checklist

Tool Scores

• run_evalB
  3.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It discloses key behaviors: simulation via the LLM, no audio cost, scoring dimensions, and output format. It doesn't mention side effects or parameter-driven choices, but the main execution model is clear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The single sentence is information-dense and front-loaded with the main action. Every clause adds value: simulation, cost, scoring dimensions, and report style.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description explains high-level behavior but omits parameter semantics and usage context. With no output schema and no annotations, the agent will struggle to know valid inputs or what N represents.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description mentions none of the three parameters (assistantId, market, vertical). It doesn't explain what market or vertical mean for the evaluation, leaving the agent to guess.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: it tests the agent via simulated scenarios, scores behavior on specific criteria, and returns a /100 report. This distinguishes it from sibling tools like list_agents or deploy_agent.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool instead of alternatives like regression_gate or audio_forensics. The description implies evaluation use but doesn't state prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• prompt_diffA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses basic behavior for each action (e.g., snapshot captures into local history, rollback pushes an old prompt back), but it does not mention side effects, reversibility, or permissions. This is adequate but lacks depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured, with a single introductory phrase followed by three short, explanatory definitions for each action. Every sentence adds value and the format is easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the core action list but omits essential context for the version parameters, output/return values, and potential side effects. Given 4 parameters, no annotations, and no output schema, the description leaves significant gaps for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, and the description only explains the 'action' enum values, which are already defined in the schema. The parameters 'assistantId', 'fromVersion', and 'toVersion' are not described at all, leaving the agent to infer their purpose and format.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: version and diff assistant prompts, with explicit actions (snapshot, diff, rollback) and what each does. This distinguishes it from sibling tools that focus on listing agents, deploying, or running evals.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage through the action semantics (e.g., snapshot to capture, diff to compare, rollback to revert), but it does not provide explicit guidance on when to choose this tool over alternatives or any exclusions. The context is clear but not fully directive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• audio_forensicsA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of disclosing safety and side effects. It describes downloading and analyzing files, which suggests a read-only operation, but it doesn't explicitly state that no modifications occur, what permissions are required, or potential side effects like data transfer or storage. The description is action-focused but lacks necessary behavioral caveats.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences that efficiently convey the tool's actions and its automation value. Every word contributes to understanding, with no redundancy or unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has a simple interface with one parameter and a clear action sequence, but the description omits details about the output format, prerequisites, or edge cases. It is adequate for basic understanding but leaves gaps that would be particularly problematic given the absence of annotations and output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has one required parameter, 'callId', with 0% schema description coverage. The description mentions 'a call' but does not explicitly link it to the 'callId' parameter or explain how to obtain it. This provides minimal added meaning beyond the parameter name.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with specific verbs (download, run, locate) and identifies the resource (3 recording tracks of a call). It distinguishes itself from sibling tools, which are focused on agent management and evaluation, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context by stating it automates the manual audio investigation, implying use when investigating noise in call recordings. It doesn't explicitly mention alternatives, but since sibling tools are unrelated, the usage context is sufficiently clear; however, there is no 'when not to use' exclusion.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• regression_gateA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and it does disclose the core behavior: it invokes run_eval, compares to a baseline, and returns allow/deny. It does not mention side effects, authorization requirements, or edge-case behavior such as equal scores, leaving some behavioral ambiguity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short, front-loaded, and every sentence adds value. It uses two crisp sentences to convey the tool's domain, behavior, and return value without unnecessary filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description provides the essential high-level purpose and return value, but no output schema and sparse parameter guidance leave important gaps for correct invocation. It does not describe the exact response structure or clarify market/vertical semantics, so the overall context is only moderately complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 25%, so the description must compensate, but it only hints at assistantId via "current assistant" and baselineScore via "stored baseline." It does not explain market or vertical, so an agent would not know their intended values or role in the evaluation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific purpose: it runs run_eval against the current assistant, compares against a baseline, and blocks the update on regression. It clearly distinguishes itself from the sibling run_eval by adding the gate/blocking behavior and the allow/deny outcome.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  "CI for prompts" immediately establishes the intended context, and the blocking behavior makes it clear this is a gate rather than a plain evaluation tool. However, it does not explicitly mention alternatives, such as using run_eval directly when a non-blocking evaluation is needed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• deploy_agentA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the behavioral burden. It discloses that the tool creates a Vapi assistant with prompt, voice, transcriber, and keywords baked in, and that one call yields one working agent. However, it does not mention potential side effects like costs, auth requirements, or whether it overwrites existing agents, which is important for a deploying/mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three short sentences that are front-loaded with the core action. The first sentence states what it does, the second details what is created, and the third summarizes the outcome. Every sentence earns its place with no redundancy or padding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, no output schema, and 6 parameters, the description covers the main purpose, markets, verticals, and baked-in components. However, it omits any indication of return values (e.g., an agent ID), potential overwriting behavior, or prerequisites. This is a notable gap for a deployment tool with moderate complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is low (33%), so the description needs to compensate. It adds enumerated values for market (fr, us, sv) and vertical (elagueur, plombier, tree-service, etc.), and references keywords/voice. Yet it leaves name, company, and voiceIdOverride unexplained beyond what the schema barely provides, so the description only partially compensates for the coverage gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Deploy') and clearly identifies the resource ('full production phone agent') with explicit market and vertical parameters. It is easily distinguished from sibling tools like list_agents, run_eval, or prompt_diff, which perform different functions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context on when to use this tool: deploying a production agent for a specific market and vertical. It implicitly differentiates from sibling tools by naming the deployment action. However, it does not explicitly state when not to use it or mention alternatives, which keeps it from a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_agentsA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. It discloses the scoped scope ('on this account') and fields returned, but doesn't explicitly state it's read-only or free of side effects. 'List' implies it, but a direct statement would make it fully transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the action and resource, with no wasted words. The field list is compact and useful.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with no parameters and no output schema, the description fully covers what it returns and why to use it, including cross-references to sibling tools. No gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so there is nothing to explain. The description adds value by listing output fields, which is more than sufficient for a no-param tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists Vapi assistants with specific attributes (name, id, model, voice, transcriber, creation date), distinct from sibling tools which perform actions like deploy, eval, or forensics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly positions this as the starting point to find assistantId for run_eval, prompt_diff, and regression_gate, providing clear when-to-use context. It doesn't mention exclusions or alternatives, but the directive is strong.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.