Server Quality Checklist

Tool Scores

• get_customers_by_countryA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and discloses key behaviors: case-insensitive matching and that unknown countries return 0 rather than an error. This is valuable, though it omits details like return format or potential rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with zero waste. The main purpose is front-loaded, followed by two crucial behavioral notes and a usage example. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple single-parameter tool, this is complete: it explains what it returns, how the parameter behaves, and the edge case for unknown countries. The lack of an output schema is acceptable since the return type ('count') is implied, and no nested objects or enums exist.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides only a bare string parameter with no description (0% coverage). The description adds meaning by explaining case-insensitivity and the handling of unknown countries, which goes beyond what the schema offers.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Returns' and the resource 'how many customers are registered in a given country', making the tool's purpose unambiguous. It is distinguishable from siblings like 'get_top_countries_by_customers' by focusing on a single country count rather than ranking.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit usage context with the example 'how many customers are from Germany?' and clarifies the unknown-country behavior. However, it does not mention when to use alternatives (e.g., ranking tools) or when not to use this tool, though the clear example partially compensates.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_top_countries_by_customersA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the sorting order (descending by customer count, ascending by country for ties) and the default limit behavior. This goes beyond a simple 'returns top countries' and gives specific behavioral details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, both essential. The first sentence states the ranking logic, the second explains the default and usage. No filler words; information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a simple tool with one optional parameter and no output schema. The description covers the ranking order, tie handling, default behavior, and a usage example. There is no missing information an agent needs to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Although the schema has no descriptions (0% coverage), the description explains the 'limit' parameter explicitly: default to 1, higher limit yields a top-N leaderboard. This adds meaning beyond the schema's numeric constraints (min, max, default) and clarifies the parameter's role.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the precise action: ranks countries by number of registered customers in descending order, with ties broken by country name. It also explicitly connects to a question ('which country has the most customers?'), distinguishing it from sibling tools that rank by spend or orders.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description specifies the default behavior (returns single top country) and how to get a top-N leaderboard by passing a higher 'limit'. It gives a concrete use case ('Use this to answer...'). It does not explicitly mention alternatives or when not to use it, but the purpose is clear enough for an agent to select it appropriately.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_top_customers_by_spendA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full responsibility for behavioral disclosure. It transparently covers exclusions ('cancelled' orders), tie-breaking (customer id ascending), interval semantics (half-open UTC, YYYY-MM-DD, open bounds), and rejection conditions (from must be earlier than to). This level of detail goes well beyond typical descriptions and leaves little ambiguity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but every sentence adds essential information. It front-loads the core ranking logic, then systematically covers edge cases (exclusions, tie-breaking, defaults, intervals) and ends with a concrete usage example. There is no fluff or repetition; the structure logical and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description comprehensively explains behavior and parameters, but it does not specify the return format (e.g., what fields are returned for each customer). Since there is no output schema, the agent may not know if the result includes customer IDs, names, or just the spend amount. It also does not explicitly state that the interval applies to order dates, though this is reasonably inferred. These minor gaps prevent a perfect score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must fully compensate. It explains 'limit' with default and usage (single top spender vs top-N), and thoroughly documents 'from' and 'to' including format, inclusivity, open-bound behavior, and ordering constraints. Every parameter is meaningfully described, adding significant value beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Ranks customers by total money spent'), defines the metric precisely (sum of quantity * unit_price), and clearly differentiates from the sibling get_top_customers_by_orders by focusing on spend rather than order count. It also provides a concrete usage phrase ('who is the customer who spent the most money?'), making the tool's purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear when-to-use via the example question, and explains parameter adjustments and interval semantics, but it does not explicitly contrast with sibling tools like get_top_customers_by_orders or state when NOT to use this tool. The metric distinction is implied, but not spelled out, so the agent has to infer the appropriate alternative.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_database_schemaA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It clearly states the tool is a listing operation (read-only) and enumerates exactly what is returned (tables, columns, types, keys). It does not mention side effects, but none are plausible for a schema query; the transparency is adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no filler. The first sentence front-loads the core functionality and enumerates what is listed; the second gives usage priority. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With zero parameters and no output schema, the description must fully specify both the return content and usage context. It does: lists the table names, column attributes, foreign keys, and explicitly directs to use it first. Nothing needed to call the tool correctly is omitted.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool takes no parameters and the schema is empty. The description explicitly says 'Takes no parameters.' Per the baseline rule for zero-parameter tools, a score of 4 is appropriate; no additional semantic explanation is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description is a specific verb-resource pairing: 'Lists every business table...' naming tables, columns, and foreign keys. It clearly differentiates from sibling tools, which are all specific analytical queries, by describing a schema-wide introspection operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs 'Use this first to understand what data is available before calling the other tools.' This tells the agent when to use it relative to all siblings and sets expectation of a preliminary step.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_top_categories_by_revenueA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It transparently explains the metric calculation, exclusion of cancelled orders, default limit, half-open interval semantics, date format, open-bound behavior, and rejection condition. This is comprehensive and leaves little ambiguity about the tool's side effects or constraints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single dense paragraph that front-loads the core purpose and then adds necessary details like exclusion, defaults, and date semantics. Every sentence contributes essential information without repetition or fluff. It is concise yet thorough, striking an ideal balance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (3 parameters, no output schema, no nested objects), the description covers all necessary operational details: sorting order, tie-breaking, exclusion, defaults, date range behavior, and validation. An agent would have everything needed to call the tool correctly. The return format is implied by the ranking nature of the tool, so no further elaboration is required.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It fully explains the 'from' and 'to' parameters, including format, half-open semantics, and validation. For 'limit', it states 'Defaults to the top 3 categories', which strongly implies that limit controls the number of returned categories, but it does not explicitly say 'limit parameter'. Given the detailed handling of the other two parameters, this is a minor gap, hence a 4 rather than a 5.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: ranking product categories by revenue with a precise formula (sum of quantity * unit_price), ordering, and tie-breaking. It also provides a concrete example use case ('top 3 product categories by revenue'). This distinguishes it from sibling tools like get_top_selling_products (which ranks products, not categories) and get_revenue_for_period (which likely returns a total, not a ranking).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly provides a usage example: 'Use this to answer "what are the top 3 product categories by revenue?"'. It does not explicitly contrast with sibling tools or state when not to use it, but the clarity of purpose makes the intended use clear. The exclusion of cancelled orders and date range handling also give context for when this tool is appropriate, though alternatives are not mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_revenue_for_periodA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses cancellation exclusion, zero-revenue success behavior, half-open interval semantics, date format, optionality of each bound, and the rejection condition when 'from' is not strictly before 'to'. This is exceptionally transparent for an unannotated tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured paragraph: the core action leads, followed by exclusions, edge-case behavior, interval semantics, and an example. Every sentence adds value, and the formatting of dates and bounds is precise without fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given a simple two-parameter tool with no output schema or annotations, the description covers parameter semantics, behavioral edge cases, and a concrete use case. It omits nothing needed to call the tool correctly. The return value (a sum) is implicitly numeric, so no further clarification is required.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%; both parameters are just strings with no description. The description fully explains that both are 'YYYY-MM-DD' dates, optional, define a half-open interval, and that 'from' must precede 'to'. This adds all needed meaning, going far beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear action: 'Sums revenue (quantity * unit_price) across all orders in the requested period', with specifics on exclusions and interval semantics. It is distinct from siblings like 'get_top_customers_by_spend' or 'get_top_selling_products', all of which target different aggregates. The resource is unambiguous and the verb 'sums' is precise.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a concrete example ('Use this to answer "how much revenue did we generate in 2025?"') and explains when the tool applies (with or without bounds). It does not explicitly state when not to use it or name alternatives, but given sibling scope, no overlapping tool exists. This is clear usage context without exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_top_customers_by_ordersA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral burden. It discloses exclusions (cancelled orders), tie-breaking (customer id ascending), default limit, half-open interval interpretation, validation (from must be earlier than to), and the effect of omitting bounds. This is exceptionally transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is detailed but every sentence adds necessary information: ranking logic, exclusions, tie-breaking, defaults, interval semantics, validation, and intended usage. It is front-loaded with the core purpose and maintains a logical flow without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description covers all essential aspects for correct invocation: what the tool does, how parameters behave, edge cases, and validation. An agent could call this tool reliably with only the provided description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must fully explain parameters. It does: 'from' and 'to' are defined as YYYY-MM-DD with half-open interval semantics and omission behavior, and 'limit' is explained with default and purpose. This exceeds what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'ranks' with the resource 'customers by how many orders they placed', making the purpose immediately clear. It also implicitly distinguishes from siblings like get_top_customers_by_spend by specifying the metric (orders).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly provides a use case ('which customer placed the most orders?') and explains the parameter behavior including defaults and interval semantics. It does not explicitly name alternative tools, but the unique metric and interval handling give clear context for when to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_top_selling_productsA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and delivers: cancelled orders are always excluded, half-open UTC interval semantics, open-bound behavior when one bound is omitted, and the from<to validation rule. This is exemplary behavioral disclosure for a read-only reporting tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Front-loaded with the core ranking behavior, then each subsequent sentence earns its place: tie-breaking, cancellation exclusion, default, interval semantics, bounds, validation, and a usage example. No filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Complete for a simple tool with zero required parameters and no output schema. Covers behavior, param semantics, defaults, validation, and usage. The implied return (ranked products with units sold) is sufficient for the stated question.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, yet the description fully documents from/to: the YYYY-MM-DD format, the half-open interval, open-side behavior, and the strict ordering validation. It also states the limit default of 5. It compensates comprehensively for a schema that lacks property descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb, resource, and metric: 'Ranks products by units sold, descending'. Defines tie-breaking (revenue then product id), distinguishing it from revenue-, customer-, and country-based siblings. Ends with an explicit usage example that pins the purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides an explicit when-to-use signal: 'Use this to answer "what are the top 5 best-selling products?"'. However, it does not name alternative siblings or state when NOT to use it, leaving the differentiation to the metric description rather than an explicit exclusion.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.