Server Quality Checklist

Tool Scores

• list_modelsC
  2.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. The 'Behavioral Transparency' section thoroughly covers side effects (none), authentication (none required for basic), rate limits (10/day free, unlimited pro), error handling, idempotency, and data privacy. This is exemplary disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Structured into sections (Behavior, When to use, Args, Behavioral Transparency) which is helpful, but it is verbose and contains repetition (e.g., side effects stated twice). The inclusion of analysis/classification content that does not align with the tool's name adds unnecessary length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers behavioral traits well but lacks clarity on what the tool actually returns (output schema exists but not described). Misses differentiation from siblings and does not explain how optional parameters affect the result. Overall, it is incomplete for a tool with 3 optional parameters and no schema descriptions.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so description must compensate. The 'Args' section gives one-line descriptions for each parameter, but they are generic (e.g., 'The filter provider to analyze or process') and do not explain how they affect the listing of models. The parameter names are somewhat self-explanatory, but the descriptions add little value and are misleading due to the purpose inconsistency.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence clearly states 'List all available models with capabilities and pricing,' but the subsequent 'When to use' section describes it as 'structured analysis or classification of inputs against established frameworks,' which is a different use case. This inconsistency undermines purpose clarity. No differentiation from sibling tools like cost_estimator or route_request.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides generic 'When to use' and 'When NOT to use' sections, but they are vague and conflict with the tool's name. Does not reference sibling tools or provide specific guidance on when to choose list_models over alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cost_estimatorA
  3.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries full burden and excels: it explicitly states the tool is read-only, stateless, idempotent, and details rate limits, authentication, error handling, and data privacy. This is comprehensive behavioral disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections but contains redundancy. For example, 'Behavior' and 'Behavioral Transparency' cover similar ground. It is longer than necessary but still organized.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (4 params, output schema exists), the description covers behavior, rate limits, and privacy well. However, parameter semantics are weak, and the description does not explicitly connect the parameters to cost calculation. It is mostly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%. The 'Args:' section in the description provides generic descriptions like 'The prompt tokens to analyze or process' for each parameter, adding minimal meaning beyond the schema. It fails to specify how these parameters relate to cost estimation (e.g., token counts for input/output).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with 'Estimate cost for a specific request across models,' which clearly states the tool's purpose with a specific verb and resource. It distinguishes from siblings like get_gateway_stats, list_models, and route_request, which serve different functions (statistics, model listing, request routing).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes 'When to use' and 'When NOT to use' sections, but the 'When to use' is overly generic ('structured analysis or classification') and not tightly aligned with cost estimation. It does not mention alternatives among siblings, so the guidance is only moderately helpful.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_gateway_statsA
  4.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully covers behavioral traits. It details read-only and stateless nature, authentication, rate limits, error handling, idempotency, and data privacy. This exceeds the minimum and provides comprehensive guidance.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections and bullet points. It front-loads the core purpose. However, it is lengthy for a simple tool with one parameter, and some information repeats (e.g., idempotency in both 'Behavior' and 'Behavioral Transparency'). Could be slightly more concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the low parameter count and presence of an output schema, the description is highly complete. It covers all relevant aspects: behavior, when to use/not use, error handling, rate limits, privacy, and idempotency. No critical gaps remain for an AI agent to invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The parameter 'api_key' is described as 'The api key to analyze or process,' adding meaning beyond the schema (which only has a title and default). However, further details (e.g., format accepted) are embedded in the behavioral transparency section, not the param-specific line. Schema coverage is 0%, so the description compensates adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence clearly states 'Get gateway usage statistics.' This is a specific verb and resource. However, the 'When to use' section describes 'structured analysis or classification of inputs against established frameworks or standards,' which seems broader and may not directly align with the tool's name and primary function, introducing slight ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description includes explicit 'When to use' and 'When NOT to use' sections, providing clear context. However, the 'When to use' is generic and does not distinguish this tool from siblings like cost_estimator or list_models. It also lacks direct comparisons to alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• route_requestA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite no annotations, the description includes a comprehensive 'Behavioral Transparency' section detailing side effects (none), authentication needs, rate limits, error handling, idempotency, and data privacy. This fully compensates for the lack of annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections and front-loaded purpose. However, the 'Args' section is somewhat wordy and could be more concise. Overall, every section adds value and the structure aids readability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (6 parameters, 1 required, no enums, output schema exists), the description covers all key aspects: purpose, usage, behavioral traits, and parameter meanings. It also includes error handling and privacy details, making it highly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The 'Args' section provides a one-line description for each parameter, but these descriptions are generic and repetitive (e.g., 'The task to analyze or process.'). They do not explain valid values, constraints, or how parameters affect routing. With 0% schema coverage, more detailed semantics would be beneficial.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The title and description clearly state the tool's purpose: routing AI requests to optimal models based on task, cost, speed, and compliance. This distinguishes it from sibling tools like cost_estimator, get_gateway_stats, and list_models.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit sections for 'When to use' and 'When NOT to use', offering clear guidance on appropriate contexts and limitations, such as not suitable for real-time production without human review.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.