Server Quality Checklist

Tool Scores

• costsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full transparency burden. It discloses detail levels with output characteristics (e.g., 'Compact total + top-N breakdown table (~20 lines)'), default parameter values, anomaly detection behavior, and date formatting requirements. It does not mention error handling or access permissions, but for a read-only query tool the disclosed behavioral traits are substantial.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with bolded section headers, uses imperative bullet lists, and packs information efficiently into about one screen. No filler—every section addresses a distinct concern (purpose, usage, detail levels, defaults, examples, date format).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has a moderate complexity level with 5 parameters and an output schema. The description covers use cases, alternatives, detail-level semantics, defaults, and examples, which is sufficient for an agent to select and invoke correctly. The presence of an output schema offsets the need to enumerate return fields, and the examples give concrete invocation patterns.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It resolves ambiguity by specifying defaults for all optional parameters (detail_level, group_by, top_n, end_time) and providing concrete examples that illustrate parameter use. The statement that 'summary (default)' and 'top-N' clarify top_n's role. The allowed group_by values are only partially specified (line_item and project_id), but examples cover typical usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Query OpenAI dollar-amount spend data,' clearly identifying the verb and resource. It distinguishes from siblings by explicitly naming the 'usage' and 'cost-comparison' tools in the DO NOT USE section, making its purpose unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description has dedicated 'USE THIS TOOL FOR' and 'DO NOT USE THIS TOOL FOR' sections. It lists specific use cases (total spend, daily breakdown, month-to-date forecast, anomaly detection) and explicitly directs users to alternative tools for token counts and month-over-month comparison, satisfying both when and when-not guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• usageA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses defaults (detail_level='summary', bucket_width='1d', top_n=10, end_time=today), service types, detail levels, and date format. However, it does not explicitly state that the operation is read-only or mention potential limitations like pagination, though the word 'Query' strongly implies a non-mutating operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with headers, bullet lists, and example calls. It front-loads the purpose, then logically presents usage instructions, service types, detail levels, defaults, and examples. Every sentence contributes value, and the format is scannable without unnecessary padding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 9-parameter tool with no schema descriptions and no annotations, this description covers all key aspects: when to use, when not to use, service types, detail levels, defaults, examples, and date formatting. The output schema exists (per context signals), so return-value details are not required in the description. This is a comprehensive and self-contained guide.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, so the description must compensate. It does so by listing service types, detailing detail levels, explaining defaults, and providing examples that illustrate parameter usage (e.g., 'group_by="project_id"' for project breakdown). However, not every parameter (e.g., project_ids, top_n) receives an explicit definition, though their meanings are inferable from context and examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Query OpenAI token and request usage data by service type' with a specific verb and resource. It also distinguishes itself from sibling tools by explicitly saying 'DO NOT USE THIS TOOL FOR: Dollar-amount costs (use 'costs' tool instead)'. This makes the purpose unambiguous and differentiates it from alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit 'USE THIS TOOL FOR' and 'DO NOT USE THIS TOOL FOR' sections, listing alternative tools ('costs' and 'cost-comparison') for excluded use cases. It also includes concrete examples of when to call the tool (e.g., 'GPT-4o usage this month'), making the usage context crystal clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cost-comparisonA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It transparently explains what the tool returns (total spend, per-line-item comparison, biggest movers), the constraint that it compares full calendar months, and includes examples. This is comprehensive behavioral coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized for a tool with 4 parameters and multiple usage scenarios. It is well-structured with clear sections (USE, DO NOT USE, PARAMETERS, OUTPUT, EXAMPLES), front-loaded with the main purpose, and every section earns its place without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 4 parameters, no schema descriptions, and no annotations, but the description compensates fully. It includes parameter details, output structure, examples, and distinctions from sibling tools. This is a complete and self-sufficient description for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description must compensate. It does so thoroughly by explaining each parameter's format (YYYY-MM), default values, allowed options for group_by ('line_item', 'project_id', or both), and top_n semantics. The description adds substantial meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Compare OpenAI costs between two months to identify spending changes.' It uses a specific verb and resource, and explicitly distinguishes itself from sibling tools 'costs' and 'usage' by naming them in the DO NOT USE section.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit USE THIS TOOL FOR and DO NOT USE THIS TOOL FOR sections, including specific alternative tools ('costs' for single-month analysis, 'usage' for token/request data). This gives clear when-to-use and when-not-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.