Server Quality Checklist

Tool Scores

• set_prompt_and_voiceB
  3.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already mark destructiveHint=true, so the mutation nature is known. The description adds one useful non-destructive detail: it preserves other model settings and attached tool IDs, which helps an agent understand scope. It does not explicitly warn about overwriting the selected fields, but that is implied by 'update'. Since it adds the preservation nuance without contradicting the destructive hint, a 3 is appropriate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence that is front-loaded with the action and resource, then adds a key caveat. No wasted words, perfectly scannable for an agent.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 7 parameters, low schema coverage, a destructive annotation, and no output schema, the description is too thin. It lacks parameter-level clarification, does not mention any side effects or limitations (beyond preservation), and provides no guidance on when to prefer this over sibling tools. An agent would struggle to use it correctly without opening the schema or experimenting.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 14% (only clientSlug has a description). The description mentions 'first-message, system-prompt, model, and voice fields' but does not map these to the actual parameter names (firstMessage, systemPrompt, modelName, modelProvider, voiceId, voiceProvider). It also fails to clarify the distinction between modelName/modelProvider and voiceId/voiceProvider. With low schema coverage, the description should compensate but does not.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb ('Update') and names the affected resources (first-message, system-prompt, model, voice fields). It also includes a distinctive preservation clause ('while preserving the assistant's other model settings and attached tool IDs') that sets it apart from a full overwrite. However, it does not explicitly contrast with any sibling tool like apply_template, so differentiation is implied rather than stated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear scenario: you want to update only these fields without touching others. This is an implicit usage condition. But it does not name alternatives (e.g., apply_template) or state when not to use this tool. The guidance is adequate but not explicit enough to confidently route an agent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• provision_clientA
  3.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare idempotentHint, destructiveHint=false, and openWorldHint. The description adds context by stating 'Reuses an existing local record by default', which clarifies the idempotent behavior, and 'secret-free' adds a security attribute. No contradiction with annotations, and it enriches the agent's mental model beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no filler. The primary action is front-loaded, and the 'reuse' behavior is stated succinctly. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 10 parameters and no output schema, the description is brief. It does not explain return values, error scenarios, or how this step fits with sibling tools like attach_phone or set_prompt_and_voice. The scope of 'provision' relative to those separate setup steps is ambiguous.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema covers 70% of parameters with descriptions. The description refers to the 'inbound answering template', aligning with templateId's default, but does not add syntax or details for the remaining parameters (e.g., crm, webhookUrl, transferNumber). It offers minimal supplementary meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear verb ('Create') and specific resources (client's Vapi assistant, local client record) and mentions the source template. It is distinct from a generic 'process' but does not explicitly differentiate from the sibling apply_template, which may also involve templates.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus siblings like apply_template, attach_phone, or set_prompt_and_voice. The note 'Reuses an existing local record by default' is an implementation detail, not a usage directive. No exclusions or alternative routes are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• apply_templateA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare destructiveHint, idempotentHint, and openWorldHint. The description adds valuable behavioral context: it explains the PATCH operation, the deep-merge last behavior for overrides, and the rendering of a local template. It does not contradict the annotations; rather, it enriches them with implementation details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences front-load the primary purpose and follow with the key behavioral nuance. No filler or redundant repetition of the title or schema. Every word adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description does not mention return values or error behavior. It also omits prerequisites (e.g., that the assistant must already exist), though the phrase 'existing Vapi assistant' hints at it. Annotations cover idempotency and destructiveness, but for a mutation tool, additional context on side effects (e.g., what changes are persisted) would improve completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers 67% of parameters with descriptions. The description clarifies the deep-merge semantics for assistantOverrides, which adds meaning beyond the schema's vague reference to a 'Vapi REST request union'. However, templateId remains unexplained, and clientSlug's description is already sufficient. Partial compensation for the coverage gap, but not complete.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (render and PATCH) and the resource (existing Vapi assistant). It goes beyond the title by specifying the local template and client settings, making the purpose concrete. However, it does not explicitly contrast with sibling tools like provision_client, so it stops short of full differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implicitly conveys when to use the tool (when you need to apply a template to an existing assistant), but it never states explicit conditions, exclusions, or alternatives. Given the sibling set includes related tools, explicit guidance would help, but the implication is reasonably clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• attach_toolsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already flag destructive behavior and non-idempotency. The description adds useful context by noting that model configuration is preserved and listing supported tool payload types. It does not explain destructive replace semantics, but the annotations carry much of the safety burden.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no fluff. The purpose is front-loaded, and the second sentence provides practical scope without unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool that can replace or merge attached tool IDs and create arbitrary tool definitions, there is no mention of return values or the destructive implications of the replace flag. The description is adequate at a high level but leaves meaningful operational gaps for the agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 50%, so the description adds meaning by clarifying that existingToolIds attaches existing IDs and toolDefinitions accepts transfer, callback, calendar, CRM, API-request, and MCP payloads. This helps compensate for the sparse schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action: 'Create reusable Vapi tools and/or attach existing tool IDs' while explicitly noting that the current assistant model configuration is preserved. This distinguishes it from sibling tools like attach_phone and set_prompt_and_voice.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage: use this when you need to add or attach Vapi tools without changing the model configuration. However, it does not explicitly state when not to use it or mention alternative sibling tools for related workflows.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• set_structured_outputsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations (destructiveHint: false, idempotentHint: false) already signal safety and non-idempotency. The description adds the key behavioral detail that omitting definitions yields a specific template schema, which is beyond what annotations convey. It doesn't discuss side effects like whether existing outputs are replaced, but the openWorldHint softens this gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, with the core action front-loaded. The first sentence names the operation and target; the second explains the default behavior without redundancy. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 3 params and no output schema, the description covers the essential behavior but omits practical details like whether calling it multiple times accumulates or replaces outputs, or whether the client must already be provisioned. Given the tool's moderate complexity and sibling context, a bit more context on side effects would be valuable, though not critical.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 67% schema coverage, clientSlug and replaceLocalIds have descriptions, but outputDefinitions (the most complex parameter) only has an opaque item description. The tool description compensates by enumerating the template fields ('name, number, reason, callback window, transfer outcome'), directly informing agents what a default output would contain. This adds genuine meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Create') and resource ('structured outputs attached to the client assistant'), and goes further to describe the default template behavior when no definitions are provided. This level of specificity distinguishes it from sibling tools like attach_tools or set_prompt_and_voice without requiring the agent to open the schema.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool by explaining the default template behavior, which gives context for when callers might omit outputDefinitions. However, it doesn't explicitly state when not to use it or name alternatives among sibling tools, leaving the agent to infer usage relative to set_prompt_and_voice or attach_tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_client_statusA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The annotations already cover readOnly, idempotent, openWorld, and non-destructive behavior. The description adds valuable extra behavioral context by disclosing that partial remote failures are returned in errors rather than failing the whole call. This goes beyond what the annotations alone communicate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences carry all the essential information, with the main action and resources front-loaded and the error-behavior detail placed second. Every sentence earns its place with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description still communicates the key returned data areas: client record, Vapi assistant, bound phone, latest call, and errors. It is sufficient for a single-parameter read operation, though a bit more detail about the shape of the errors field would remove remaining ambiguity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and the single parameter clientSlug is fully documented in the schema with format and pattern details. The description does not add meaning beyond the schema, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific read operation and the exact resources being read: local client record, current Vapi assistant, bound phone, and latest call. This clearly distinguishes it from the mutation-oriented siblings like provision_client, attach_phone, and set_prompt_and_voice.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies its use by saying 'Read' and by listing status-like data, but it does not explicitly state when to choose this over a sibling or when not to use it. There is no alternative tool mentioned, so the agent must infer the usage context from the sibling names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• smoke_test_callA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description goes beyond annotations by disclosing that the call is real, potentially billable, and requires explicit confirmation, which complements the destructiveHint. It also mentions optional polling and returns, adding behavioral context not covered by structured fields.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences with the core action and key constraints front-loaded. It contains no fluff or repetition and is entirely on-topic.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool that places a call, the description covers the essential points: billable, confirmation needed, and optional return of transcript/data. It doesn't clarify parameter interactions or output format, but the schema and annotations fill some gaps. Given no output schema, the mention of return data is helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 60% (3/5 parameters have descriptions). The description does not add parameter-specific information beyond what's already in the schema; it only indirectly references confirmation. The undocumented parameters (waitSeconds, phoneNumberId) are not explained in the description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (place an outbound call) and the resource (client assistant), and distinguishes it from sibling provisioning/config tools. It also adds critical context (billable, confirmation required) that sets it apart.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies a smoke-testing use case from the title but gives no explicit guidance on when to use this tool versus alternatives. It does not mention when not to use it or name any sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• attach_phoneA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds behavioral detail beyond annotations, such as the enforced 'client assistantId' and the ability to either bind an existing ID or create a new one. This provides insight into side effects, though it does not detail conflict handling or error cases.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that front-loads the primary action (bind or create/import) and avoids unnecessary detail. It is well-structured and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the schema (with property descriptions) and annotations (destructive, non-idempotent), the description provides sufficient operational context. There is no output schema, so return values need not be explained. Minor gaps exist, such as behavior when both parameters are provided, but these are not critical.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description clarifies the roles of phoneNumberId (existing ID to bind) and phoneConfiguration (body for creation/import), adding meaningful semantics beyond the schema's generic descriptions. It explains how the parameters relate to the two operational modes.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: to bind an existing Vapi phone-number ID or create/import a number via a provider-specific body. It distinguishes from sibling tools like attach_tools, which likely handle tools rather than phone numbers, making its role unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by describing two modes (bind existing or create/import) but does not explicitly compare with alternatives like attach_tools or provision_client. It gives conditional guidance but lacks clear 'when to use this vs. that' instructions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.