Server Quality Checklist

Tool Scores

• rank_countries_by_customersA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses meaningful behavior beyond a bare summary: countries are derived from phone-number prefixes, there is no country column, results are returned as {country_code, country_name, customer_count}, and the limit has both a default and a maximum clamp. This is especially valuable because no annotations are provided. It does not explicitly state read-only behavior, but that is strongly implied for a ranking operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and every sentence earns its place: the operation, the derived-country caveat, the return shape, and the limit behavior are all included without filler. It is front-loaded with the action and directly clarified important nuances immediately.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read-style ranking tool with no output schema, the description is largely complete: it explains the ranking, the derivation logic, the returned fields, and the parameter behavior. It is slightly incomplete only in not explicitly addressing potential ambiguity with a similarly named sibling or clarifying whether the result includes all countries above a zero threshold.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter, limit, receives no schema description, so the description must carry the parameter semantics. The description covers this well: it explains the default (100) and the maximum clamp (1000), which is information not present in the input schema beyond the parameter name and nullability.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action: ranking countries by customer count in descending order. It also makes the unusual derivation explicit (country comes from phone-number prefix, not a country column). However, it does not explicitly differentiate from sibling tool count_customers_by_country, so the agent may not know which tool to choose when the sibling name is available.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The intended use is implied by the description: use this tool when you need a ranked list of countries by customer count. However, the description gives no explicit guidance about when to choose this tool over similar sibling tools like count_customers_by_country, nor does it state any exclusions or alternative conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• revenue_by_categoryA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It clearly states the order status filter, the exact revenue formula using quantity times unit_price, the output shape, and pagination behavior. It omits explicit darting direction, though 'Rank' reasonably implies descending order.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded, stating the core purpose in the first sentence and then covering inputs, filters, computation, return shape, and pagination in two more short sentences. Every sentence contributes essential information without filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only reporting tool with two optional parameters and no output schema, this is nearly complete. It explains the join, the relevant for the filter, the exact aggregation, the return list structure of {category, revenue}, and pagination. The only real missing piece is an explicit ordering clause, but the word 'rank' likely covers it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema gives no parameter descriptions, so the description must compensate. It does clarify that `limit` defaults to 100 with a maximum of 1000 and that `offset` paginates, giving real semantic value beyond the raw schema. It could be more explicit that offset is the number of rows to skip, but it is largely sufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Rank product categories by revenue' and explicitly names the join path through orders -> order_items -> products. This is enough to distinguish it from sibling tools like top_products (product-level) and revenue_by_year (time-based), even though it does not name them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  There is no explicit when-to-use or when-not-to-use guidance. It does specify that only 'completed' and 'shipped' orders count, which is useful filtering context, but it does not tell the agent how to decide between this and closely-related alternatives such as revenue_by_year or top_products.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_tableA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Although there are no annotations, the description discloses the exact return shape ({table, columns, foreign_keys}) and the scope of information retrieved, leaving no ambiguity that this is a read-only introspection endpoint. It does not cover error behavior for invalid table names or format details of returned values, but it is sufficiently transparent for a metadata lookup.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, with the core purpose in the first sentence, usage guidance in the second, and the return shape and parameter enumeration in the third. Every clause earns its place, so it is both compact and highly informative.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter metadata tool with no output schema, the description covers everything an agent needs: what the tool returns, how to call it, valid inputs, and when to use it. No significant contextual gaps remain that would prevent correct invocation or understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides only a string named 'table' with no description or enum, giving an agent no guidance. The description fully compensates by describing what the parameter is ('the table name') and enumerating the four valid values (customers, products, orders, order_items). This is exactly the value a tool description should add beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description has a specific verb ('Describe'), a well-defined resource ('one table'), and enumerates exactly what schema details are returned (columns with name/type/not_null/default/primary_key, plus foreign keys). It clearly distinguishes itself from the sibling tools, which are list/analytical tools rather than schema introspection.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives concrete guidance on when to use it: 'Use this before building any mental model of how tables relate.' It also limits usage by naming the only valid table values. It does not explicitly say when NOT to use it in favor of a sibling, but the context is clear enough that an agent can route correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_tablesA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral disclosure burden and does state the return shape: a list of {name, description}. It clearly communicates that the operation is a read-only listing of all tables, which is sufficient given the tool's simple intent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately brief and front-loads the core action and scope before explaining return shape and usage. Every sentence adds distinct value, with no repetition or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter catalog tool with no output schema, the description alone fully covers what the tool does, what it returns, and when to use it. Nothing critical is missing, and the tool is simple enough that additional detail would not meaningfully improve agent performance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and the schema coverage is 100%, so there is no parameter information the description needs to supplement. The description focuses on the return format instead, which is the only semantic detail an agent needs for invocation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the action as listing every table in the shop database and clarifies that each entry includes a short description of the table's contents. It also positions the tool as an entry point for discovering available entities, distinguishing it from sibling tools like describe_table.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises using this tool first to understand what entities are available, providing clear contextual guidance for when it should be called. It does not explicitly name alternatives or exclude cases, but for a zero-parameter discovery tool that guidance is sufficient.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• count_customers_by_countryA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral burden, and it does this thoroughly. It discloses how countries are derived, the exact return shape, the 'honest 0' behavior for valid countries without customers, and the error condition for unknown country names.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded with the most important fact first: the country is not a literal column. Every sentence adds distinct value covering semantics, parameter behavior, output, zero-case handling, and errors.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool with no output schema and no annotations, the description fully explains input semantics and output format. No essential detail is missing for an agent to correctly select and invoke the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, but the description more than compensates by explaining the only parameter's meaning: `country` may be absent/null, a full country name, or an ISO alpha-2 code. It also maps each form of the parameter to the resulting behavior.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the specific action ('Count customers by country') and resource clearly, and it explicitly distinguishes the semantic scope by saying the country is derived from the E.164 phone-number prefix rather than being a column. This prevents confusion with table-oriented siblings such as list_tables and describe_table.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives precise guidance for the two usage modes: no argument returns all country counts, and a country argument filters to one country. It also warns against the likely mistake of looking for a `country` column. It does not explicitly name alternatives like rank_countries_by_customers, so it stops short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• revenue_by_yearA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of disclosure, and it succeeds: it defines the exact aggregation, the status filter, and the edge cases. Specifically, it states that a year with no qualifying orders returns 0 with a note instead of substituting a different year, and that invalid years return an error.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three dense sentences with no redundant framing. Every sentence adds distinct value: return semantics, aggregation logic, and edge-case behavior. The most central result ('total revenue for a single year') is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a one-parameter query tool with no output schema or annotations, the description is complete: it specifies the request semantics, the denominator filter, the zero-result case, and the failure case. An agent can predict behavior without any other documentation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must explain 'year' beyond 'integer' it does. The example '2025', the '4-digit integer' constraint, and the invalid-year error behavior give an agent the validation and expected input semantics it needs.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource ('Return total revenue for a single year') and gives the exact computation semantics (SUM(orders.total_amount), order_date, status filter). This scoping distinguishes it from the sibling revenue_by_category and from the customer/country tools without needing to open the schema.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description makes clear that this tool is for single-year revenue queries and unobtrusively implies the contrast with revenue_by_category. It does not explicitly name alternatives or give when-not-to-use conditions, but the single-year scoping gives agents enough context to select it appropriately.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• top_customersA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral disclosure burden, and it does this very well: it states exactly which order statuses count as earned for 'spend', which statuses are excluded, how 'order_count' treats cancelled orders, what fields are returned, and the limit/offset behavior. This goes well beyond what the schema provides.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but well organized, with a front-loaded purpose, clear bullet-style sections for each mode, and no filler. Every sentence communicates a useful constraint or behavior.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 3-parameter tool with no output schema, the description is complete: it defines ranking semantics, status filtering, returned customer fields, the two allowed modes, limits, and pagination. An agent has enough information to call the tool correctly without guessing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must fully explain the parameters. It explains both valid values of `by`, the exact computation for each, the effect of `limit` with its default of 100 and cap of 1000, and the pagination role of `offset`. This fully compensates for the empty schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific verb ('Rank customers') and resource (customers) and immediately distinguishes the two ranking modes: total spend vs number of orders. It is clearly differentiated from siblings such as top_products and rank_countries_by_customers.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives concrete usage context: how to choose by='spend' versus by='order_count', which statuses count for each mode, and how limit/offset behave. It does not explicitly name sibling alternatives or say when not to use this tool, but the conditional semantics are clear enough for selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• top_productsA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full behavioral burden and does so thoroughly. It discloses the order-status filter, that money is computed from order_items.unit_price rather than current products.price, that ranking changes with metric, and the exact result fields.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Five dense sentences, each serving a distinct purpose: the core action, status filter, revenue derivation, result shape, and pagination defaults. The most important information is front-loaded and there is no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given there is no output schema and no annotations, the description is complete: it states result structure, metric-dependent ranking behavior, critical data source rule, status filtering, and limits. Minor assumptions like descending order are natural from the concept of ranking.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate—and it does. It defines the metric options and behaviors, clarifies the default limit of 100 and maximum of 1000, and explains offset pagination. This goes well beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource: 'Rank products by units sold or revenue.' It clearly distinguishes what this tool computes from sibling tools like top_customers and revenue_by_category, without any ambiguity about scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides clear operational usage context: valid metric values, default and max limit, pagination, and which order statuses are included. It does not explicitly name sibling tools or state when to prefer an alternative, so it stops short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.