Server Quality Checklist

Tool Scores

• count_topups_by_numberA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool counts top-ups and returns a count or error message, but lacks details on permissions, rate limits, error conditions, or data freshness. For a tool with zero annotation coverage, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the core purpose, followed by clear sections for arguments and returns. Every sentence earns its place by directly supporting tool understanding, with no redundant or verbose language, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (1 parameter, no output schema, no annotations), the description is adequate but incomplete. It covers the basic purpose and parameter semantics but lacks behavioral details like error handling or usage guidelines. For a simple query tool, it meets minimum viability but has clear gaps in contextual richness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaningful context for the single parameter 'phone_number' by explaining it is 'the phone number to count transactions for,' which clarifies its role beyond the schema's basic type and title. With schema description coverage at 0%, the description effectively compensates by providing semantic value, though it could include format examples (e.g., E.164).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Counts') and resource ('number of top-ups for a specific phone number'), distinguishing it from siblings like 'check_balance' (balance checking), 'get_last_topups' (retrieving recent transactions), 'load_airtime' (airtime loading), and 'sum_last_n_topups' (summing recent transactions). It precisely defines what the tool does without ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention scenarios where counting top-ups is preferred over other tools like 'get_last_topups' or 'sum_last_n_topups', nor does it specify prerequisites or exclusions. Usage is implied only by the purpose statement, lacking explicit context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_balanceA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes the tool's behavior: connecting to an external API, fetching application data, and returning either a balance message or an error. It covers the core operation and error handling, though it lacks details like rate limits, authentication requirements, or specific error conditions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, starting with the core purpose. The second sentence adds necessary detail about the API connection, and the 'Returns' section clarifies output behavior. It's efficient with minimal waste, though the 'Returns' formatting could be slightly more concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (0 parameters, no output schema, no annotations), the description is reasonably complete. It explains what the tool does, how it operates via API, and what it returns. For a simple balance check tool, this covers essential context, though it could benefit from more detail on error scenarios or API specifics.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so the schema fully documents the lack of inputs. The description adds value by confirming no parameters are needed ('checks the airtime balance' implies a simple query without inputs), aligning with the schema. Baseline is 4 for 0 parameters, as it provides clear semantic context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Checks') and resource ('airtime balance of the Africa's Talking account'), distinguishing it from sibling tools that focus on top-up operations rather than balance inquiry. It precisely defines what the tool does without ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by mentioning it fetches 'application data' and 'account balance', suggesting it's for checking current balance status. However, it doesn't explicitly state when to use this tool versus alternatives like checking transaction history through sibling tools, nor does it provide exclusion criteria or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• load_airtimeA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses key behaviors: formatting phone numbers, using Africa's Talking API, and logging to a database. However, it lacks details on error handling, rate limits, authentication needs, or what specific data is logged.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the core purpose in the first sentence. Each subsequent sentence adds necessary detail without redundancy. The Args/Returns sections are clearly formatted and efficiently convey parameter and return value information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a mutation tool with no annotations and no output schema, the description does a good job covering purpose, parameters, and basic behavior. However, it lacks details about the return message format, error conditions, or database logging specifics that would be helpful for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It provides clear semantic meaning for all three parameters: phone_number identifies the recipient, amount specifies airtime quantity, and currency_code defines transaction currency with an example ('KES'). This adds significant value beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('sends airtime', 'logs the transaction') and identifies the resource (phone number). It distinguishes from sibling tools like check_balance or get_last_topups by focusing on sending rather than querying.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for sending airtime via Africa's Talking API, but provides no explicit guidance on when to use this tool versus alternatives like count_topups_by_number or sum_last_n_topups. No prerequisites, exclusions, or comparison to siblings are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_last_topupsA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes the tool as retrieving transactions and returning a formatted string or message, which covers basic behavior. However, it lacks details on error handling, data format specifics, or performance aspects like rate limits or permissions needed, leaving gaps in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, starting with a clear purpose statement followed by structured sections for arguments and returns. Every sentence adds value without redundancy, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (1 optional parameter, no output schema, no annotations), the description is mostly complete. It covers purpose, parameter semantics, and return behavior. However, it could improve by addressing potential edge cases or linking to sibling tools for better context, slightly reducing completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaningful context for the single parameter 'limit,' explaining it as 'the number of recent transactions to fetch' with a default of 3, which goes beyond the input schema's basic type and title. Since schema description coverage is 0%, the description compensates well by providing clear semantics for the parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verb ('retrieves') and resource ('last N top-up transactions from the database'). It distinguishes from siblings by focusing on recent transactions rather than checking balances, counting by number, loading airtime, or summing amounts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by specifying 'last N top-up transactions,' suggesting it's for recent transaction review rather than other operations like checking balances or loading airtime. However, it does not explicitly state when to use this tool versus alternatives like 'sum_last_n_topups' or 'count_topups_by_number,' missing explicit exclusions or comparisons.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• sum_last_n_topupsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and discloses key behavioral traits: it only sums successful top-ups, retrieves from a database, ensures currency consistency, and handles error cases (mixed currencies or no transactions). It doesn't cover aspects like performance, rate limits, or authentication needs, but provides substantial operational context beyond basic functionality.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded: the first sentence states the core purpose, followed by operational details, then clearly formatted Args and Returns sections. Every sentence adds value—none are redundant or vague—and the structure enhances readability without unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (database query with calculations and error handling), no annotations, and no output schema, the description is largely complete: it covers purpose, parameters, behavior, and return values. It could benefit from mentioning prerequisites (e.g., authentication) or performance characteristics, but adequately addresses core functionality and error cases for an agent to use it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaningful semantics beyond the input schema: it explains that 'n' represents 'the number of recent top-ups to sum' with a default of 3, and clarifies it's optional. With 0% schema description coverage (schema only has type and title), the description fully compensates by providing clear parameter meaning and usage context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('calculates', 'retrieves', 'ensures') and resources ('last n successful top-ups', 'transactions from the database'). It distinguishes from siblings like 'check_balance' (current balance), 'count_topups_by_number' (counts by number), 'get_last_topups' (lists without summing), and 'load_airtime' (different operation).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for summing recent top-ups, but doesn't explicitly state when to use this tool versus alternatives like 'get_last_topups' (for listing) or 'count_topups_by_number' (for counting by number). It mentions currency consistency as a requirement, which provides some contextual guidance, but lacks explicit when/when-not scenarios or named alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.