Server Quality Checklist

Tool Scores

• cost_from_usageA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description discloses cache multipliers and date defaults. However, it does not mention potential rate limits, caching behavior, or whether the tool is read-only. The 'shared cache multipliers' hint at internal state but are not fully explained.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no wasted words. The key purpose is front-loaded, followed by essential details about token fields and date default.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the nested tokens object and absence of output schema, the description covers token field meaning and date defaults. It could mention the return value format (e.g., a number in USD), but the function name makes it clear.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds value by listing the specific token fields (input, output, cache_read, etc.) and explaining cache multipliers, which goes beyond the schema's property descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'Value a token rollup in USD at a point in time', which is a specific verb and resource. It distinguishes from sibling tools like 'current_price' (which likely prices a single token) and 'price_on' (which may fetch historical prices) by focusing on token rollup valuation with cache multipliers.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool (when you have token counts to value) but does not explicitly guide when not to use it or contrast with siblings. It mentions defaults but lacks exclusion criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compareA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description adds key behavioral info: each row resolves independently and unknown IDs are reported per row. It does not explicitly state read-only nature, but it's implied and sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with purpose, no wasted words. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description lacks detail about the return format (e.g., table structure). It covers behavior but not output shape, leaving some incompleteness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline 3. The description adds meaning beyond schema by explaining independent resolution and unknown ID handling for 'models', and 'omit for today' for 'date'.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool provides side-by-side input/output prices for multiple model IDs on a date, distinguishing it from siblings like 'current_price' (single model) and 'list_models' (listing only).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says it's for comparing several models, implying use when a comparison is needed. It does not explicitly mention alternatives or when not to use, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• current_priceA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description carries full burden. It discloses that prices are today's and from first-party source, but does not explicitly state read-only behavior or other traits like rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no redundancy, and the core function is front-loaded. Every word adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 params, no output schema, no annotations), the description fully covers purpose, unit, and parameter usage. Complete for effective selection and invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds meaningful context: model accepts short/aliased ids, provider disambiguates shared bare ids. This goes beyond the schema's minimal descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns today's input and output price per million tokens for a model id, specifying unit and data source. It distinguishes from sibling tools like 'price_on' by focusing on current prices.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains how to use parameters (accepts aliased ids, provider for disambiguation), but does not specify when to use this tool versus alternatives or when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_modelsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses the output (model IDs, provider, aliases) and the optional filtering behavior. No side effects or limitations are mentioned, but the tool is simple and read-only.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: first states the core functionality, second states its purpose. No wasted words; front-loaded and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple listing tool with one optional parameter and no output schema, the description covers the essential aspects: what it returns, filtering, and how it relates to sibling tools. Slightly vague about 'dataset,' but adequate given the context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with a description for 'provider.' The tool description adds context by explaining the effect of the filter and that each result includes provider and aliases, going beyond the schema alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it lists model IDs known to the dataset, optionally filtered by provider, and includes provider and aliases. This distinguishes it from sibling tools like compare, cost_from_usage, current_price, and price_on, which focus on cost or comparison.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises to 'discover valid ids for the other tools,' providing clear context for when to use it. Lacks explicit when-not-to-use or alternatives, but the purpose is well-stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• price_onA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description properly explains the function (lookup, no side effects), clarifies the meaning of 'covered=false', and indicates the output includes the price and source. It is transparent about what the tool does without contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (3 sentences) and front-loaded with the core purpose. Each sentence adds essential information: the output, use cases, and the special case of 'covered=false'. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite lacking an output schema, the description explains the key output fields (price per million tokens, source, covered flag). For a simple lookup with 2 required parameters, this is sufficient to understand what the tool returns.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, baselining at 3. The description adds value by explaining the purpose of 'model' (id or alias), 'date' (YYYY-MM-DD), and 'provider' (disambiguation), and clarifies the meaning of 'covered=false' in the output.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns the price per million tokens in effect on a given date, specifically for point-in-time lookup to value past usage or see price changes. It distinguishes from siblings like 'current_price' and 'compare' by emphasizing historical lookup and noting the 'covered=false' flag.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises to use it for 'valuing past usage' or 'seeing how a price changed over time', implying it's not for current prices. Although alternatives are not named, the sibling tools list provides context, making the usage reasonably clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.