Server Quality Checklist

Tool Scores

• get_commitment_utilizationA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full disclosure burden. It mentions one behavioral trait – returning an explanatory message when no commitments exist – which is useful. However, it does not cover other behaviors like data granularity handling, boundary effects, authentication requirements, or whether the operation is read-only. It is not misleading, but only partially transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences with no fluff. The core purpose is front-loaded, and the edge-case note is placed second. It is efficient and easy to consume, though slightly longer than necessary.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 5 parameters, 3 enums, no output schema, and no annotations, the description covers the main purpose and one edge case but omits usage guidance, parameter clarification for granularity, and any detail on the return structure. It is adequate for basic selection but not fully complete for correct invocation without additional schema inspection.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is low (40%), and the description adds only minimal parameter context. It names the commitment types (savings_plans, reservations) which maps to the commitment_type enum, and mentions utilization vs coverage which echoes the report field. But it does not explain granularity or the start/end date format beyond the schema, and it fails to compensate for the 60% of parameters lacking schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Get'), a precise resource ('Savings Plans or Reserved Instance utilization and coverage'), and a time-bound scope. It clearly differentiates from siblings like cost/usage/forecast/anomalies/budgets/tags tools, so an agent can tell what this tool addresses without opening the schema.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for commitment utilization/coverage tracking, but gives no explicit guidance on when to choose it over alternatives like get_cost_and_usage or get_cost_forecast. No exclusions or when-not-to-use context is provided, leaving the decision to the agent's inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_cost_and_usageA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It adds date inclusivity ('Start date is inclusive, end date is exclusive') and says grouping/filtering are supported, but these largely repeat schema descriptions. It omits output shape, pagination, error behavior, rate limits, or any constraints on the query window, which leaves meaningful behavioral gaps for a 6-parameter tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, front-loaded with the core purpose, then capabilities, then usage context. Each sentence earns its place: the first identifies the resource and action, the second conveys grouping/filtering power, and the third gives both a date-semantics detail and a clear 'when to use' signal. No fluff or repetition beyond acceptable reinforcement.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema and no annotations, the description must compensate by explaining what the tool returns and any operational limits. It covers what the tool does but not the structure/format of its response, potential result sizes, or failure modes. For a tool with 6 parameters and nested object schemas, this is a significant completeness gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds a few examples of valid group_by keys (SERVICE, USAGE_TYPE, REGION, LINKED_ACCOUNT, or cost allocation TAG), but these overlap with the schema's existing enum/description text. It does not meaningfully explain metrics, granularity trade-offs, or filter syntax beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb ('Query') and resource ('AWS cost and usage for account 135709585800') and highlights key capabilities: grouping, filtering, and time period. It also positions itself as the primary tool for spend analysis, which distinguishes it from siblings like get_cost_forecast, get_cost_anomalies, and describe_budgets.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear use-case context via 'This is the primary tool for spend analysis: monthly trends, per-service breakdowns, daily spikes, etc.' This tells the agent when to reach for this tool among siblings, but it does not explicitly state when not to use it or name alternative tools for non-spend-analysis needs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_cost_anomaliesA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds useful behavioral context: it returns an empty list when no anomaly monitors are configured, and includes root causes and impact. With no annotations provided, the description carries the behavioral burden, but does not disclose max_results handling, pagination, permission requirements, or error cases.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no redundant content. The main action and scope come first, the edge case behavior is appended second, avoiding waste. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a relatively simple list tool with only 3 parameters. The description states what is returned, the date-range behavior, and an important edge case. The main gap is that it does not describe the output shape in more detail or the meaning of max_results, but the schema partially compensates.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already documents start and end dates as YYYY-MM-DD and marks them required; the description does not go beyond that. It does not explain max_results, which is the least-documented parameter, so the description leaves a modest gap with no added parameter meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') and a distinct resource ('cost anomalies detected by AWS Cost Anomaly Detection'), and adds what the result contains ('root causes and impact'). This clearly differentiates it from sibling tools like get_cost_and_usage or get_cost_forecast.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The intended use case is implied: call this when you want anomalies found by AWS Cost Anomaly Detection in a date range. It does not explicitly say when not to use it or name alternatives, so the agent must infer that from the description and sibling names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_dimension_valuesA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral disclosure burden. It adds useful context by noting that values are those that 'actually incurred cost' in the time period, which is non-obvious. However, it does not disclose output format, pagination, or other runtime behavior, leaving some gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no filler. The core purpose is front-loaded, and the usage guidance is compactly appended. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a list-style tool with no output schema and no annotations, the description covers the essential context: what it lists, the time-period constraint, and how it fits into a workflow. It could mention return format or pagination, but the description is sufficient for correct invocation in most cases.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 75%, so the schema already documents most parameters. The description adds examples of dimension values (services, regions, usage types) and implies the search_string use case, but does not substantially enrich parameter meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List available values') and identifies the resource ('Cost Explorer dimension in a time period'), with concrete examples of dimensions. It also distinguishes itself from the sibling get_cost_and_usage by framing itself as a prerequisite discovery step.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool: 'Use this to discover exact value strings before filtering in get_cost_and_usage.' This provides clear context and a workflow connection, though it does not enumerate when not to use it or mention other sibling alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_cost_forecastA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral burden. It clearly states that the tool returns a mean forecast plus an 80% prediction interval, which sets expectations about results and implies a read-only forecasting operation. Additional error or edge-case behavior is not disclosed, but the core output behavior is well covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact, clearly worded, and front-loaded. Every sentence adds value — first the action and date constraint, then the exact shape of the forecast output. No wasted words or irrelevant background.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a forecast tool with a small parameter set, the description adequately explains the main purpose, expected result, and important date caveat. It lacks deeper detail on unusual inputs or failure behavior, but the essential information for selecting and simple invocation is present.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 60%, and the tool description does not compensate: it restates the start-date constraint already present in the schema but gives no added meaning for metric, granularity, or the filter object. The description names no metrics, default choices, or range semantics beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource: 'Forecast future AWS spend for the account over a date range' — which clearly identifies the action and target. It also names the output type (mean forecast plus 80% prediction interval), making the tool easy to distinguish from cost/usage retrieval tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The instruction 'must start today or later' establishes a clear constraint and scenario for when this tool should be used. It does not explicitly mention alternatives like get_cost_and_usage for historical spend, but the future-focused wording makes the intended context fairly obvious.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_cost_allocation_tagsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description correctly shoulders the behavioral burden. It clearly discloses the conditional behavior (with/without tag_key), the time-period scope, and the downstream consumer of the returned keys. It does not describe pagination or whether duplicate values are collapsed, but the core behavior is transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no filler: action, resource, optional parameter behavior, and downstream integration are all covered. Front-loading the core purpose allows an agent to immediately route correctly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a modest read-style list tool with no output schema and no annotations, this definition is largely complete: it covers the two possible return intentions, the time period, and the consumer of the output. A little return-format detail or note about pagination would make it fully bulletproof, but nothing essential is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 67%, and the description adds critical semantics beyond the schema for tag_key: how providing it changes the result, and how omitting it lists all available tag keys. The date-scope for start/end is handled by the schema, so the description does not need to repeat it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names the exact resource (cost allocation tag values/keys), states the verb (List), and clarifies the two different output modes based on tag_key. It clearly separates this from billing query tools by referencing group_by type=TAG.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives clear, actionable usage context, including the exact dependent tool and grouping mode ('Use this keys with group_by type=TAG in get_cost_and_usage'). It does not explicitly explain when not to use this tool or contrast it with get_dimension_values, so it stops short of explicit exclusion guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_budgetsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden of behavioral disclosure. 'List' signals a read-only operation and the description states exactly which data attributes will be returned. It does not detail potential failure modes or pagination, but for a parameterless list operation this is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single efficient sentence that front-loads the action and resource while packing in the key return details. There is no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a parameterless listing tool with no output schema, the description fully covers what the tool does and what information it returns. An agent can correctly decide to call it and understand the expected result.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and no schema description, so the baseline for parameter semantics is 4. The description does not add parameter-related detail because none is needed to invoke the tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb ('List') and a concrete resource ('AWS Budgets configured for the account'), and it enumerates what is returned: limits, actual spend, and forecast. This clearly differentiates it from sibling tools focused on cost usage, forecasts, anomalies, and tags.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for retrieving an account-level view of budgets with spend and forecast data, which gives clear usage context. It does not explicitly name alternatives or in which cases to prefer another sibling, but the resource focus makes the intended use apparent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.