Server Quality Checklist

Tool Scores

• get_productsA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It does disclose the default sort ('Sorted by id by default') and the DB-validation constraint on category, which are genuinely useful. However, it does not state the return format, whether all products are returned when no filters are applied, or how limit/offset interplay with the default ordering — notable gaps for a tool with zero annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three tight sentences with zero filler. The core purpose is front-loaded in the first clause, and each remaining sentence earns its place by clarifying validation and filter semantics. Perfectly sized for a filtered-list tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having six parameters and no annotations or output schema, the description is largely complete for correct invocation: all filter semantics are covered and the default ordering is disclosed. The only genuine gap is the absence of any statement about returned fields or no-parameter behavior, but for a simple list tool these are reasonably inferable, so the definition is only mildly incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3 with the schema bearing the load. The description adds marginal value: it clarifies that category is validated against existing DB categories and restates the inStock and price filter logic already present in the schema. It does not fully compensate beyond that, but the schema already documents all six parameters adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb+resource: 'List products.' This clearly identifies the tool as the product-listing operation and distinguishes it from siblings like get_customers and get_orders by entity type. The analytic siblings (top_products_by_quantity, etc.) are also implicitly set apart, though the description does not explicitly contrast against them.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains how each filter behaves (category validation, inStock stock_quantity > 0, min/max price), which gives implicit context on how to narrow a product query. However, it never states any when-to-use or when-not-to-use guidance relative to siblings such as top_products_by_quantity or revenue_by_period, leaving the selection between raw listing and analytic tools to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_order_itemsA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral disclosure burden. It usefully reveals that each result includes the product name via a LEFT JOIN to products and that results are sorted by id by default. It does not cover response shape or default result size, but for a read-only list tool the disclosed behavior is reasonably transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three compact sentences: purpose, filters, join behavior, and default sort. Every sentence contributes useful information and there is no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, filtering, joining, and sorting, but there is no output schema and no description of the overall return shape beyond the product name. It also does not state what happens when no filters are supplied, which would be useful for an agent deciding whether to provide a required filter.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the input schema already documents limit, offset, order_id, and product_id. The description only restates the filtering semantics and adds no new parameter-level detail beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific verb and resource: 'List order line items.' It clearly distinguishes this from sibling tools like get_orders or get_products by focusing on line items rather than top-level orders, though it does not explicitly name or contrast any sibling.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool: for listing line items, optionally filtered by order_id and/or product_id. However, it provides no explicit guidance about when not to use it or which sibling alternative might be preferable, such as get_orders or the top_* analytics tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• revenue_by_periodA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full disclosure burden and handles it well: it reveals the cancelled-order exclusion, ascending sort order, the 'empty period yields no row' gap behavior (valuable — prevents assuming zero-filled periods), and the exact row shape { period, revenue }. Minor gaps remain (behavior for inverted from/to ranges, timezone handling of ISO datetimes), but the core behavioral traits an agent needs are disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three dense sentences with zero filler: sentence one states the computation and exclusion, sentence two explains the only nontrivial parameter's format, sentence three covers output shape, sort order, and gap behavior. Every clause earns its place and the most decision-relevant fact (cancelled exclusion) is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a filtered aggregation tool with no output schema, the description is fully self-sufficient: it specifies input parameters (via schema), grouping options with default, return row structure, sort order, and missing-data behavior. An agent has everything needed to invoke it correctly and interpret results without additional guessing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% — all three parameters (to, from, group_by) are fully described in the schema, including enum values and defaults. The description's group_by explanation ('year' default, 'month' -> YYYY-MM) largely restates schema content rather than adding new meaning; baseline 3 is appropriate since the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific aggregation: 'Revenue (sum of orders.total_amount) EXCLUDING cancelled orders, grouped by period.' The verb is implicit but unmistakable, the resource (orders) and computation (sum of total_amount) are precise, and the cancellation filter plus time-grouping clearly separates it from entity-breakdown siblings like top_categories_by_revenue and raw-data siblings like get_orders.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Usage context is implied by the name and grouping semantics — an agent can infer this is for revenue-over-time analysis and that entity-level revenue alternatives exist among siblings. However, no explicit when-to-use or when-not-to-use guidance is given; the description never names top_categories_by_revenue or top_customers_by_spend as the entity-breakdown alternatives, so routing relies entirely on inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• top_customers_by_spendA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden of behavioral disclosure. It clearly discloses a meaningful filtering behavior (excluding cancelled orders), an optional date-range filter, and the default limit of 10. This is strong coverage for a read-only ranking tool, though it does not mention ordering direction or potential tie-breaking.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two dense sentences with no filler. The core computation and exclusion rule are front-loaded, followed by the optional filter, output shape, and default. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description gives enough to call and understand the tool correctly: metric definition, exclusion rule, optional filters, output row shape, and default limit. It lacks only minor details like explicit sort order and whether 'orders' is a count, which are largely inferable from context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds value beyond the schema by tying the date parameters to order_date and explicitly stating the default limit of 10, which the schema does not mention. It also clarifies that all filters are optional.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific ranked list resource ('top customers by total spend'), defines the metric as sum of orders.total_amount, and explicitly excludes cancelled orders. The row-shape preview also clarifies exactly what is returned, and the metric clearly distinguishes it from siblings like top_products_by_quantity or customers_by_order_count.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description makes the tool's scope clear enough to infer when to use it, but it does not explicitly state when to prefer it over sibling tools such as customers_by_order_count or revenue_by_period. No exclusions or alternative-routing guidance is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• top_categories_by_revenueA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. It discloses key behaviors: the revenue calculation, the exclusion of cancelled orders, and the default limit of 10. It also specifies the output row structure. It does not mention error handling or performance, but for a read-only query this is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence that front-loads the purpose, then provides the formula, exclusion, output format, and default limit. No fluff or repetition; every element earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite lacking an output schema, the description fully specifies the row shape ({ category, revenue }), the default limit, the metric definition, and the filtering rule. An agent can invoke the tool and interpret results correctly without additional information. Sibling tools are clearly differentiated by name and purpose.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds value by stating the default limit (10) and clarifying that 'revenue' means the sum of quantity * unit_price, which aids interpretation of the 'to'/'from' parameters in the context of the calculation. This goes beyond the schema's terse descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the exact output: top product categories by revenue, with a precise formula (sum of order_items.quantity * unit_price) and exclusion of cancelled orders. It clearly distinguishes from siblings like top_customers_by_spend and revenue_by_period by the specific resource (categories) and metric (revenue).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage (when you need top categories by revenue) but does not explicitly state when not to use it or mention alternatives. No exclusions or comparisons to siblings are given, so the agent must infer from context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• customers_by_order_countA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly warns that all orders are counted, including canceled ones, and documents the output row shape and default limit. It could additionally mention sorting order or pagination behavior, but the most critical behavioral quirk is explicitly called out.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded: it states the core purpose first, then adds the critical caveat about canceled orders, then the optional filtering and output details. Every sentence contributes meaningful information without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a relatively simple analytic tool with no output schema, the description covers the essential elements: metric, output row format, a critical counting caveat, optional filters, and default limit. It does not explicitly state the sort order, but 'Top customers' reasonably implies descending order by order count.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already covers all three parameters with 100% description coverage, so the baseline is 3. The description adds value by specifying that from/to filter on order_date and by revealing the default limit of 10, which is not present in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb-resource combination: 'Top customers by number of orders placed,' which clearly identifies what the tool computes. It also distinguishes itself from sibling analytic tools by explicitly noting that canceled orders are included, making its scope clear relative to alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context on when to use the tool by contrasting it with 'other analytic tools' regarding canceled orders, which helps an agent avoid misusing it for exclusion-based analyses. It also mentions optional date filtering, but it does not explicitly name an alternative or state exact conditions for when to prefer a different sibling tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_customersA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It discloses nontrivial behavior: case-insensitive substring matching across specific fields, and pagination metadata including total ignoring limit. It is clearly a read-oriented list operation, so no contradiction or hidden mutation risk is apparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two compact sentences front-load the core purpose, then cover search semantics and return shape. Every clause earns its place; there is no filler or repetition of the schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is largely sufficient for a simple list tool: it covers invocation, search fields, and top-level return metadata. However, no output schema exists and the description does not specify the shape of an individual customer row beyond calling it a row, leaving some ambiguity about available fields.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already documents all three parameters with 100% coverage, so baseline is 3. The description adds value by specifying case-insensitive substring semantics and clarifying that total counts all matches regardless of limit, going beyond the schema's LIKE %term% phrasing.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource ('List customers') and clarifies the tool's scope with optional search behavior. It clearly distinguishes from sibling aggregation tools like top_customers_by_spend by describing a paginated flat list rather than a ranking or metric.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly implies usage: list customers, optionally narrowed by a free-text search. It does not explicitly name alternatives or give when-not-to-use guidance, but the pagination and search context are enough for an agent to recognize when this tool applies.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_ordersA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral disclosure burden. It adds meaningful specifics beyond the schema: status values are validated against the DB, results are sorted by id by default, cancelled orders are included here, and `include_items` attaches an items array. It doesn't mention default limit/offset behavior or error cases, but the disclosed behaviors are substantial for an unannotated tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is four tight sentences with no filler. The main verb and resource are in the first sentence, filters follow logically, and each sentence earns its place by adding distinct useful information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only list tool with 7 optional parameters and no output schema, the description covers the core behaviors: what is listed, how to filter, how to include items, default ordering, and how cancelled orders are treated. It does not describe the full order object shape or pagination defaults, but those are partially covered by the schema, and the description is sufficient for an agent to invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds value by grouping `from_date` and `to_date` as a date range on order_date, clarifying that ISO date or datetime strings are accepted, and reinforcing that `include_items` attaches an items array. This is extra semantic clarity beyond the individual property descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear verb and resource ('List orders') and then enumerates the specific filtering dimensions and the `include_items` attachment behavior, so an agent immediately knows what the tool does. It also differentiates itself from analytic siblings by stating that cancelled orders are shown here but excluded from analytic tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not name the sibling tools directly, but it gives clear context: this is the order-list tool with exact status/customer/date filtering and optional item embedding, and it explicitly notes that cancelled orders are included here but excluded from analytic tools. That exclusion helps an agent choose this tool over analytic siblings when cancelled data is needed, though it could be stronger by naming alternatives like get_order_items.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• top_products_by_quantityA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral burden, and it does so well: it discloses the join needed to exclude cancelled orders, the date filtering, the exact output row shape, and the default limit. No surprising behavior is hidden.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two compact sentences front-load the core metric, then supply the critical exclusion, output shape, and default. Every clause carries information; nothing is redundant.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  All optional parameters are covered, the output row is fully described without an output schema, and the behavior around cancelled orders and limit defaults is explicit. An agent has enough to call and interpret the result correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the parameter baseline is 3. The description adds one piece of schema-independent semantics (default limit 10) and clarifies that date parameters filter via order_date, slightly exceeding the baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a precise metric ('units sold'), the aggregation source (order_items.quantity), and the scope (products). It also distinguishes itself from revenue/customer siblings by stating it ranks products by quantity, making the tool's purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The computation and filters make the intended use clear: rank products by units sold within an optional date range while excluding cancelled orders. It does not explicitly list sibling alternatives or say when not to use it, but the metric and row shape leave little inferential gap.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.