Server Quality Checklist

Tool Scores

• free_tier_statusB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosing behavior. It implies a read-only status check by its name and phrasing, and adds useful context about the tool's purpose in revealing charges that are otherwise unnoticed. However, it does not explicitly confirm safety, discuss permissions, or mention any limitations or edge cases.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short and front-loaded with the core purpose in the first sentence. The second sentence adds motivational context rather than redundant detail, so it earns its place. It is not overly verbose or repetitive.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (1 optional param, no required params) and an output schema exists, so the description need not detail return values. However, the unexplained 'profile' parameter and lack of explicit behavioral notes leave gaps. It is adequate but not fully complete for an agent to use without guessing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has one optional 'profile' parameter with 0% description coverage, and the description does not mention the parameter at all. Since the description fails to compensate for the complete lack of schema documentation for this parameter, it provides no added meaning beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool reports free tier allowances and current account proximity to each limit, which is specific enough to distinguish from siblings like what_is_wasting_money or capabilities. However, it lacks an explicit verb like 'get' or 'list' and does not directly contrast with sibling tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a concrete scenario ('Not in the bill, so a charge sitting just over an allowance is invisible without it') that explains when this tool is valuable. It gives clear context for use, though it does not explicitly name alternatives or state when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• capabilitiesA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It explicitly states the tool is read-only 'by construction rather than by policy' and that the answer is short, which covers key safety and behavior aspects. The output schema handles return format details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no waste. It front-loads the purpose and adds a brief, relevant behavioral note, earning a perfect score for conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter tool with an output schema, the description adequately covers purpose and read-only nature. It could slightly expand on what 'can and cannot do' encompasses (e.g., server-level limitations vs tool-level), but overall it is sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With zero parameters, the baseline for this dimension is 4. The description adds no parameter-related meaning, which is appropriate since there are no parameters to document.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool reports what the server can and cannot do, which distinguishes it from sibling tools focused on specific analyses. However, it lacks an explicit verb like 'Lists' or 'Returns,' relying on a noun phrase, so it's clear but not maximally specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no explicit guidance on when to use this tool versus alternatives. The note about asking before granting access addresses authorization rather than usage context, leaving the agent to infer that this is a read-only capability check.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• what_was_not_examinedA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses meaningful behavioral details beyond the tool name: it explains that spend is classified into three states and that the unexamined list is ordered largest first. Since no annotations are provided, this added context helps set expectations, though it still does not explicitly mention read-only status or permissions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is intentionally brief, front-loading the core purpose in the first line and then providing the rationale and ordering detail in two supporting sentences. Every sentence adds value, with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description captures the tool's purpose and one key behavioral trait (ordering), but it omits any explanation of the optional parameters and does not offer comparative usage guidance with sibling tools. Since an output schema exists, return values need not be described, but the missing parameter semantics and absence of when-to-use guidance leave the description only partially complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema includes two optional parameters ('cur' and 'profile') with no descriptions (0% coverage), and the tool description does not mention or explain either parameter. This leaves the agent without any hint of their meaning or effect, a substantial gap that the description fails to compensate for.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'show' and a clear resource 'spend no rule looks at', effectively distinguishing it from sibling tools like what_is_wasting_money or what_changed. It further explains the denominator concept, reinforcing the purpose and making the tool's intent unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context by explaining that every dollar is either flagged, checked and clean, or never examined, clarifying the tool's role in identifying what wasn't reviewed. However, it does not explicitly state when to use this tool over alternatives or when not to use it, so it lacks explicit exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• show_fixA
  4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full responsibility for disclosing side effects. It explicitly says 'Nothing is applied' and differentiates output by resource type (unified diff for Terraform, AWS CLI command otherwise). This is critical behavioral transparency for a tool that might otherwise be assumed to apply changes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact: three sentences, front-loaded with the purpose, then two sentences of essential behavior. Every sentence adds value, with no filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the core functionality, safety profile, and output format, and an output schema exists. However, with 4 parameters and 0% schema coverage, leaving parameters unexplained creates a significant gap for an agent trying to invoke the tool correctly. It is adequate but not fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, and the description provides no explanation of the parameters. It does not mention finding_id, repo, cur, or profile semantics, leaving the agent to guess their meanings. The description fails to compensate for the missing schema documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Generate the change for one finding', a specific verb-object pair that clearly identifies the tool's function. The output form ('as a diff or a command') and the scope ('one finding') further refine the purpose, distinguishing it from sibling tools that analyze waste or changes rather than produce fixes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It states 'Nothing is applied. The caller reviews and merges,' which makes the intended use case clear: generate a proposed change for human review. However, it does not explicitly contrast this with alternatives or state when not to use it, so it falls short of a full 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• what_changedA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden for behavioral disclosure, and it does provide a key behavioral constraint: only complete billing periods are compared, and it explains the rationale (avoiding calendar artifacts). It does not cover every trait (e.g., permissions, rate limits), but the given nuance is valuable and non-obvious.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences: the first front-loads the purpose, the second adds a necessary caveat about complete periods. Every sentence earns its place, with no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The output schema likely covers return values, but the description lacks parameter semantics and any guidance on when to use this tool versus siblings like what_is_wasting_money. It provides the core purpose and a key limitation, but the operational context is incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has two parameters ('cur', 'profile') with no descriptions (0% coverage), and the description does not mention them at all. The general purpose hints at what they might mean (current period, billing profile), but the agent is left guessing, and the description does not compensate for the missing schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Explain') and resource ('a bill that moved'), and explicitly lists the key outputs (growth, services driving it, spikes). This clearly distinguishes it from sibling tools like what_is_wasting_money, which focus on waste rather than change analysis.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies a clear usage context (explaining bill changes) and provides an explicit when-not-to-use rule ('Compares complete billing periods only... a month still running is not comparable'). It does not explicitly name alternative tools, so it stops short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• what_is_wasting_moneyA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosure. It openly states what the tool returns (evidence, fix, blast radius, and consequences of false findings), which goes beyond a generic 'find' description. It also mentions credential behavior and the Cost Explorer fallback. It does not disclose performance characteristics or side effects, but for a read-oriented analysis tool, it is reasonably transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, well-structured, and front-loaded. It uses a clear summary line, a 'Start here' instruction, and a compact parameter list. No wasted words; every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity and the existence of an output schema, the description covers the core aspects: purpose, parameters, and key tradeoffs. It explains why to use Cost Explorer versus CUR and what the findings include. It could optionally mention expected runtime or AWS permissions, but given the output schema and clear context, it is reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All three parameters (profile, cur, repo) are explicitly explained in the description, with default behaviors and tradeoffs. The schema has zero descriptions, so this fully compensates. profile gets credential-chain context, cur gets the s3 URI and the Cost Explorer alternative, repo gets its purpose. This exceeds what the schema alone provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource: 'Find avoidable cloud spend on an AWS account.' It clearly outlines what is returned (cost, avoidable items, evidence, fix, blast radius, what breaks), and the phrase 'Start here' establishes its role as the entry-point tool. This distinguishes it from siblings like what_was_not_examined or what_changed, which target narrower subtopics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear directive to use this tool first ('Start here.') and explains when to use or omit each parameter: omitting cur uses Cost Explorer (no setup, but no per-resource detail), repo has a specific purpose. However, it does not explicitly compare against sibling tools or state when NOT to use it, so it falls short of a full 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.