Server Quality Checklist

Tool Scores

• ai_admin_query_costsA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare this as read-only and open-world, so the description's main contribution is noting that it queries 'normalized' cost facts and works with 'enabled' providers. This adds a bit of context but doesn't disclose behaviors like credential requirements, result shape, or rate limits, so it does not go far beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences that get straight to the point, front-loading the action and purpose without any filler. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having six parameters, nested objects, and no output schema, the description is minimal and omits important context such as how cost facts are returned, what 'normalized' entails, how multiple providers are specified, and how credentials are handled. This leaves the agent under-informed for a tool of this complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description only references provider_options, one of six parameters. The schema coverage is 0%, but the description does not explain the meaning of start, end, providers, include_raw, or credential_refs, so it fails to compensate for the sparse schema semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action and resource: 'query normalized cost facts' from 'enabled providers.' This distinguishes it from sibling tools that list providers, query usage, or fetch dashboard bundles, making the tool's purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this tool is for querying cost data and instructs how to pass provider-specific options, but it does not explicitly state when to prefer this over the sibling tools or any exclusions. There is no mention of alternatives or when-not-to-use, so guidance is only implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ai_admin_query_usageA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, so the safe read-only nature is covered. The description adds context about passing provider-specific options via provider_options[provider_id], but does not disclose rate limits, pagination, or return format.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, each earning its place. The first states the purpose; the second gives a direct usage tip. No redundant wording or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (7 parameters, nested objects, no output schema), the description is too sparse. It omits critical context about the date range, provider selection, raw data inclusion, bucket widths, credential references, and expected response shape.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description only explains provider_options. The other six parameters (start, end, providers, include_raw, bucket_width, credential_refs) remain completely undocumented, leaving the agent without guidance on their meaning or usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Query' and the resource 'normalized usage facts' from 'one or more enabled providers', which is specific and distinguishes it from sibling tools like query_costs and query_dashboard_bundle.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies its use for querying usage data but does not explicitly contrast it with alternatives or state when not to use it. No exclusions or alternative tool references are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ai_admin_query_dashboard_bundleA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and openWorldHint=true, so the safety profile is known. The description adds the provider_options structure and 'enabled providers' scope, but does not disclose output format, pagination, or behavior when no providers are specified. With annotations covering the core safety traits, a 3 is appropriate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, both front-loaded with the primary purpose then a key parameter detail. No wasted words—every sentence carries useful information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 8 parameters, nested objects, and no output schema, the description is too sparse. It explains only the provider_options nesting and the cross-provider scope. Most parameters are undocumented, and the return shape is unknown. For a tool of this complexity, the description is incomplete and leaves substantial gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds meaningful explanation for provider_options (provider_options[provider_id]) and implies providers refers to enabled providers, but leaves start, end, top_n, bucket_width, credential_refs, and include_metadata unexplained. Some value is added, but not sufficient for full compensation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'build' and resource 'dashboard bundles', with a specific scope of 'across one or more enabled providers'. It distinguishes itself from sibling tools like query_usage and query_costs by emphasizing bundling across providers.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context: this tool is for building aggregated dashboard bundles across multiple providers. It implies usage for multi-provider scenarios, but does not explicitly mention alternatives or exclusion criteria. The sibling names hint at alternatives, but the description itself lacks a when-not-to-use statement.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ai_admin_list_providersA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint: true and openWorldHint: false, so the safety profile is covered. The description adds value by disclosing the specific content returned (health, reporting surfaces, known limitations), which goes beyond the raw annotations. However, it does not describe output format, pagination, or any other behavioral traits, so it meets the baseline but doesn't exceed it.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that conveys all essential information without wasted words. It is appropriately concise and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a no-parameter, read-only listing tool, the description is fairly complete. It enumerates the included content: enabled providers, health, configured reporting surfaces, and known limitations. There is no output schema, so the description carries the burden of explaining what is returned, which it does sufficiently. A minor gap is not mentioning whether disabled providers are omitted or if any filtering is needed, but the phrase 'enabled providers' clarifies the scope.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has no parameters, and the input schema is an empty object. The baseline for 0 params is 4, and there is nothing for the description to explain. The description adds no parameter info, but none is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'List' and identifies the resource 'providers' with clear scope: enabled providers, health, configured reporting surfaces, and known limitations. This clearly distinguishes it from sibling tools like ai_admin_query_usage, ai_admin_query_costs, and ai_admin_query_dashboard_bundle, which focus on usage/costs/dashboards rather than provider status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for checking enabled providers and their health, but it does not explicitly state when to use this tool versus the alternatives (e.g., 'use ai_admin_query_usage for usage data'). There is no exclusion or alternative guidance beyond the resource name itself. The context signals from sibling names help, but the description does not articulate when this is the right choice.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.