Server Quality Checklist

Tool Scores

• estimate_costA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It mentions models and caching but does not disclose whether the tool is read-only, requires authentication, or has rate limits. For a cost estimation tool, this is adequate but not fully transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that front-loads the key action and resource. It is efficient with no wasted words, though it could be slightly restructured for clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 5 parameters, no output schema, and no annotations, the description is insufficient. It does not explain the output format (e.g., per-model breakdown), mention underlying pricing, or handle edge cases, leaving significant gaps for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters. The description adds context by naming the specific models (Haiku/Sonnet/Opus) and referencing the optional caching ratio, which goes beyond the schema details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool estimates Claude API cost for Haiku/Sonnet/Opus models based on token counts or raw text, with optional caching. This distinguishes it from siblings like batch_savings, caching_breakeven, compare_providers, and estimate_tokens.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for cost estimation but does not explicitly specify when to use it versus alternatives like batch_savings or compare_providers. There is no 'when not to use' guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• caching_breakevenA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the break-even threshold (~1.28 reuses) but does not explain the output format, side effects, or model parameter impact.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences, no fluff, key information front-loaded (verb and resource first).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple computation tool without output schema or annotations, the description covers the core purpose and a key insight. It could detail the output but remains adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 67% (prompt_tokens and reuses have descriptions). The description adds context about static prompts but does not clarify the model parameter or its effect on the calculation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Compute' and the resource 'prompt-caching savings and break-even for a static prompt reused N times', distinguishing it from sibling tools like batch_savings or estimate_cost.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for static prompts and caching scenarios but does not explicitly state when to use or avoid this tool, nor does it mention alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• estimate_tokensA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. Discloses language awareness (Korean/CJK) and that no API key is needed, but does not mention output format or computational behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences, no fluff. Front-loaded with purpose, efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one required parameter and no output schema, the description provides key context (language awareness, no auth). Lacks output format info, but still sufficient for likely use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with a clear parameter description. The tool description adds no additional semantics beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the action ('Estimate the number of tokens') and resource ('for a piece of text'). Distinguishes from siblings (cost tools) by focusing on token counting.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit comparison to sibling tools or guidance on when to use this vs alternatives. However, the use case is implied by the purpose and siblings are clearly different.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• batch_savingsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. Discloses discount percentage and use case, but lacks statements on side effects (read-only), permissions, or rate limits. Adequate for a simple calculator tool but minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences front-load purpose and usage guidance. No redundancy or unnecessary text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Lacks description of output (e.g., returns annual savings, per-day savings). With no output schema, this gap leaves agents uncertain about return value. Otherwise provides sufficient context for a simple tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 75% (3 of 4 parameters described in schema). Description adds no parameter-specific details beyond schema. Baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it computes Batch API savings (50% off input+output) for recurring workloads. Differentiates from sibling tools like caching_breakeven and estimate_cost by specifying batch discount.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'Use when work is async/non-realtime (≤24h SLA acceptable)', providing a clear condition. Does not mention alternatives or exclusions, but the guidance is direct and helpful.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_providersA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description fully bears transparency. It discloses the return format (table, sorted), the set of models (8 from 3 providers), and the purpose (compare Claude vs alternatives). It does not mention pricing source, update frequency, or limitations, but sufficiently conveys behavioral expectations for a comparison tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, zero waste. The first sentence states the core action, the second adds key output detail. Every word is purposeful and efficiently conveys the tool's value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and moderate complexity (8 models, 3 providers), the description is largely complete. It specifies the output format (table, sorted), the purpose, and the required inputs. Minor gap: does not mention that prices are assumed current, but overall sufficient for agent selection and invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description reiterates that token counts are used ('for given token counts') but adds no new parameter-level detail beyond the schema. The context of comparing across models is useful but does not enhance individual parameter semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool compares API costs across 8 models from three providers (Anthropic, OpenAI, Google) given token counts, and specifies the output (sorted table, cheapest-first, shows Claude's position). This is a specific verb+resource+scope that distinguishes it from siblings like estimate_cost (single model) or batch_savings (batch pricing).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Description implies usage for comparing costs across providers, but does not explicitly state when to use this tool over alternatives like estimate_cost (for single-model cost) or batch_savings. No when-not or exclusion criteria provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.