Server Quality Checklist

Tool Scores

• get_kpisA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already establish read-only, idempotent, safe behavior. The description adds valuable context beyond annotations: it specifies 'live pricing data' (freshness), explains the semantic meaning of each KPI (e.g., 'Output Premium: how much more output tokens cost vs input'), and clarifies the tiered access model.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with zero waste: opening sentence establishes the resource, the bullet list efficiently details the 6 return values with brief explanations, and the final sentence covers access constraints. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema exists, the description compensates effectively by enumerating and explaining all 6 returned KPIs. For a single-parameter tool with full schema coverage, detailing the return values comprehensively provides complete contextual coverage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema coverage, the baseline is 3. The description adds meaningful context by stating KPIs are 'available to all tiers,' reinforcing the schema's guidance that the API key can be omitted for free tier access. This connects the parameter to the broader access model.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns '6 key performance indicators derived from live pricing data' and names the specific resource (ATOM Inference Price Index). However, it lacks explicit differentiation from similar siblings like 'get_market_stats' or 'get_index_benchmarks'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description notes that 'KPIs are available to all tiers,' hinting at the optional API key usage, but provides no explicit guidance on when to choose this over siblings like 'get_market_stats'. The list of 6 KPIs provides implicit context but no direct when-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_vendorsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and idempotentHint=true. The description adds value by specifying return fields (vendor name, country, region, pricing page URL) and categorization schema (four channel types), compensating for missing output schema. However, it omits rate limits, pagination behavior, or detailed auth implications beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Front-loaded with purpose ('List all AI inference vendors'), followed by return value, categorization context, then examples. Every sentence earns its place. The structure progresses logically from what it does to what it returns to how to use it, with no redundant text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Adequately compensates for missing output schema by listing return fields. Covers all 3 parameters (0 required) and their optional nature. Minor gap: doesn't mention pagination, result limits, or whether the vendor list is exhaustive vs. curated, which would be useful for a 'list all' tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is met. The description adds semantic context via examples showing how 'European' maps to region and 'Chinese' maps to country, clarifying the filter intent. However, it doesn't add syntax details, validation rules, or explain the _atom_api_key behavior beyond the schema description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Specific verb 'List' + resource 'AI inference vendors' + scope 'tracked by ATOM'. Distinguishes from siblings by specifying it returns basic metadata (name, country, region, pricing URL) and spans four channel types, differentiating it from get_vendor_catalog or compare_prices.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides three concrete examples mapping natural language queries to parameter usage ('European AI vendors' → region='Europe'), effectively demonstrating when to apply filters. Lacks explicit 'when not to use' or alternative naming (e.g., doesn't say 'use compare_prices for price comparisons'), but the examples provide clear contextual guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_model_detailA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations declare readOnly/idempotent safety, while the description adds substantial behavioral context: it details the exact data structure returned (model_registry data with context window, parameters, open-source status, training cutoff, model family) and explains the pricing scope (all SKU pricing across every vendor). It also implies tiered access behavior via the API key parameter description.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is tightly structured with zero waste: first sentence establishes purpose and scope, second sentence details return payload structure, followed by concrete usage examples. Every sentence earns its place with specific technical details rather than generic filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the absence of an output schema, the description adequately compensates by enumerating the specific fields and data categories returned (registry fields, pricing SKUs). It covers the complexity of the 2-parameter input well and explains the access tier behavior, though it omits error handling scenarios (e.g., model not found).

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the baseline is 3. The description elevates this by providing concrete examples of model_name values ('GPT-4o', 'Claude Sonnet 4.5') that clarify expected input formats and naming conventions beyond the generic schema description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states 'Deep dive on a single AI model: technical specs + pricing across all vendors,' providing a specific verb (deep dive), resource (AI model), and scope (technical specs + pricing across vendors). It clearly distinguishes from siblings like search_models (implied by 'single' vs search) and compare_prices (comprehensive detail vs comparison).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The examples ('Tell me everything about GPT-4o', 'Claude Sonnet 4.5 specs and pricing') provide clear context for when to use this tool—when seeking comprehensive details about a specific known model. However, it does not explicitly name alternatives like search_models for when the exact model name is unknown.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_modelsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations declare readOnlyHint=true and idempotentHint=true. The description adds valuable behavioral context not in annotations: it discloses the tiered return behavior ('Free tier shows count + price range; paid tier shows full details') and clarifies that pricing data is included in results. No contradictions with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Structure is optimally front-loaded: scope (vendors/SKUs), capabilities (query dimensions), return value explanation, tier limitations, then concrete examples. Every sentence serves a distinct purpose. No redundant text. The arrow notation in examples ('→') efficiently maps natural language to parameters.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite lacking an output schema, the description compensates adequately by explaining what gets returned ('matching models with pricing') and detailing the tier-based response differences (free vs. paid). Given the 12-parameter complexity and 100% schema coverage, this is sufficient for an agent to invoke the tool, though slightly more detail on 'full details' content would warrant a 5.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, establishing a baseline of 3. The description elevates this by providing practical examples demonstrating parameter combinations (e.g., open_source=true with modality='Text'), which adds real-world usage context beyond the raw schema definitions. It also summarizes the filterable dimensions in natural language before listing examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb ('Search and filter') + specific resource ('AI inference models') + clear scope ('40+ vendors and 1,600+ SKUs'). It effectively distinguishes from siblings like 'get_model_detail' (retrieve specific record) and 'list_vendors' (list vendors not models) by emphasizing the search/filter capability across a broad catalog.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides three concrete, mapped examples showing query patterns ('Find open-source text models under $1/M tokens' → parameter mapping). This gives clear implicit guidance on when to use the tool. However, it lacks explicit 'when not to use' guidance (e.g., does not mention to use 'get_model_detail' when looking up a specific model by ID rather than searching).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_pricesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Adds crucial behavioral detail not present in annotations: results are 'sorted cheapest first.' Annotations already establish read-only/idempotent safety (readOnlyHint=true, destructiveHint=false), so description appropriately focuses on business logic behavior rather than safety. No contradictions with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Perfectly structured with one-line purpose, behavioral constraint (sorting), use case context (cost optimization), and three targeted examples. Every sentence serves a distinct function; no redundancy or filler content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 6 parameters with complete schema coverage and safety annotations, the description provides excellent context through examples. Minor gap: no output schema exists, and description could briefly characterize the return value (e.g., 'returns list of vendor offers') to complete the picture.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite 100% schema coverage, the examples section adds substantial semantic value by demonstrating the relationship between model_name and model_family parameters, and showing how to map user intent (e.g., 'Cheapest GPT-4 family') to specific parameter values including the direction enum.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description opens with specific verb 'Cross-vendor price comparison' and identifies the resource (prices for specific models/families). Distinct from siblings like get_vendor_catalog (which likely lists vendors without price comparison) and get_model_detail (which provides specifications rather than pricing).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides three concrete examples mapping natural language queries to exact parameter combinations, giving clear context for how to use model_name vs model_family and direction parameters. Lacks explicit 'when not to use' or named sibling alternatives, but examples provide strong implicit guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_market_statsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations declare read-only/idempotent safety; the description adds critical return value structure (vendor/model/SKU counts, price quartiles, modality breakdown) compensating for the missing output schema. Notes aggregation scope but omits data freshness or rate limiting details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Every sentence earns its place: purpose declaration, return value specification (essential without output schema), filtering behavior, and targeted examples. No redundant or filler text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 2-parameter tool with complete annotations, the description adequately compensates for the missing output schema by detailing return statistics (median, quartiles, breakdowns) and covers the free-tier limitation via the schema description. No gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema coverage establishing a baseline of 3, the description adds value by specifying that modality acts as a 'filter' (changing the aggregation scope), adding semantic meaning beyond the schema's 'focus on' language.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Aggregate AI inference market intelligence' (specific verb + resource) and distinguishes from siblings by emphasizing aggregate metrics (counts, distributions, quartiles) rather than individual model/vendor retrieval like get_model_detail or search_models.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides three concrete examples showing when to use the tool (market overview, text pricing stats, image generation stats), establishing clear context. Lacks explicit 'when not to use' guidance or named sibling alternatives, preventing a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_vendor_catalogA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations declare the operation is read-only, idempotent, and safe. The description adds valuable return value details ('vendor metadata (country, region, pricing page URL) plus every model and SKU') that compensate for the missing output schema, clarifying what data structure to expect without contradicting the safety annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with purpose, followed by return value disclosure, then practical examples. Every sentence serves a distinct function—scope definition, output specification, or usage illustration—with zero redundancy or filler text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 5 parameters including optional filters (modality, direction), pagination (limit), and authentication (_atom_api_key), plus the absence of an output schema, the description adequately covers the return structure and provides sufficient examples to infer usage patterns for the filtering capabilities.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema coverage, baseline is 3. The examples add semantic context beyond raw schema definitions by demonstrating how to translate user intents like 'OpenAI's text model pricing' into specific vendor/modality parameter values, effectively illustrating the relationship between the parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific scope ('Full catalog for a specific vendor: all models, modalities, and pricing') that clearly distinguishes this from siblings like list_vendors (which lists vendors) and get_model_detail (which retrieves single model info). The verb 'Get' from the title combined with 'catalog' precisely identifies the resource operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Three concrete examples map natural language queries to specific parameter combinations, providing clear context for how to invoke the tool. However, it lacks explicit guidance on when NOT to use this tool (e.g., 'use search_models instead for cross-vendor searches') which would earn a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_index_benchmarksA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  While annotations declare readOnlyHint=true and destructiveHint=false, the description adds substantial behavioral context: the return structure (14 indexes across 4 categories), specific pricing components included (input, cached input, output), scope limitations (market-wide vs. individual), and availability ('Fully public — available to all tiers'). Does not mention rate limits or pagination beyond the limit parameter.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is well-structured and front-loaded: opens with the AIPI definition, immediately states the 14-index return structure, breaks down the four categories with their analytical purposes, clarifies scope and availability, then provides actionable examples. Every sentence conveys unique information; no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite lacking an output schema, the description comprehensively details the return values (14 benchmarks across 4 categories with specific pricing components). Given the moderate complexity (4 optional parameters, domain-specific concepts) and presence of clear annotations, the description provides sufficient context for an agent to understand the full tool contract.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema coverage, the baseline is 3. The description adds significant value by explaining the taxonomy of index_category (detailing that Modality has 6 types, Channel has 4, etc.) and providing concrete examples for index_code in the examples section ('AIPI TXT GLB', 'AIPI OSS GLB'). This semantic context helps agents understand valid values beyond the schema's type definitions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly defines the tool as returning 'AIPI (ATOM Inference Price Index) — chained matched-model price benchmarks for AI inference.' It distinguishes from siblings by explicitly stating these are 'market-wide benchmarks, not individual vendor prices,' contrasting with tools like compare_prices or get_vendor_catalog that likely return specific vendor data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit when-to-use guidance through five concrete examples mapping natural language queries to specific parameter combinations. Also clarifies when NOT to use by stating these are not individual vendor prices, implicitly directing users to sibling tools for specific pricing queries. Includes clear intent: 'Use them to understand where the market is and how it's moving.'

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.