Server Quality Checklist

Tool Scores

• summarize_categoryA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It states the tool retrieves a 'representative sample', implying non-exhaustive results, and instructs not to invent details. However, it does not disclose sampling method, determinism, or side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no wasted words. First sentence states purpose, second gives usage guidance and constraints. Front-loaded and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given an output schema exists, description need not detail return values. However, it omits how category and intent combine (AND/OR) and how the sample is selected (random, recent, etc.). Adequate for simple use but could be more complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so description should compensate. It adds meaning for category and intent (used for filtering) but provides no explanation for sample_size (default 15) beyond the name, leaving its purpose and constraints unclear.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves a representative sample of customer messages and agent responses for a category and/or intent to allow summarization. It distinguishes from siblings by focusing on sampling for summarization rather than counting, filtering, or listing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit usage examples ('summarize the FEEDBACK category' or 'how do reps respond to cancellations') and instructs to base summary only on returned text, avoiding invention. Lacks explicit when-not-to-use or alternative tools, but context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_categoriesA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It implies a safe read-only operation but does not explicitly state that it is non-destructive or mention any limitations (e.g., no pagination, no auth requirements). The description is adequate but lacks explicit behavioral guarantees.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: one sentence for purpose and one for usage guidelines. It is front-loaded with the key action, and every word serves a purpose. No unnecessary text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (1 boolean parameter, output schema exists, sibling tools are related), the description covers the essential scenarios. It explains when to use it relative to filtering or counting. However, it could briefly mention the optional count parameter to be fully complete, but overall it is well-contextualized.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has one optional boolean parameter 'with_counts' (default false), but the description does not mention it at all. With 0% schema description coverage, the description fails to inform the agent about the ability to include counts. This is a significant omission for a single parameter tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'List the distinct customer-service categories in the dataset' with concrete examples (e.g., refund, shipping). It distinguishes from siblings like list_intents which list different entities.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly advises when to use this tool: 'Use this first when the user asks what categories exist or when you need to confirm a valid category name before filtering or counting.' This gives clear context and directs the agent to use it as a prerequisite for other operations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_intentsA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so description carries full burden. It discloses listing distinct intents and optional counts, but omits details like pagination, ordering, or performance. Adequate but not thorough for a list tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences only, front-loaded with key action, includes concrete examples. Every word earns its place with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 optional params) and existence of output schema, the description covers core functionality well. Could mention output schema for completeness, but not essential.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description compensates fully. It explains 'category' as optional restriction and 'with_counts' for distribution, giving concrete examples (e.g., 'ACCOUNT category', 'get_refund'). Adds meaning beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'List the distinct intents' with an optional restriction by category. It distinguishes from siblings by mentioning discovery of valid intent names and distribution queries, fulfilling a specific verb+resource+scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit use cases: to discover valid intent names and to get distribution with counts. Examples clarify context. Missing explicit 'when not to use' or alternatives, but the sibling list hints at other tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• filter_recordsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It discloses that returned data is a small sample, never the full set, and includes total count. Lacks details on authentication or side effects but adequately covers the key behavioral trait.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with no fluff. Key information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 4 optional parameters and an output schema, the description covers purpose, usage, and return behavior. It could mention that parameters are optional and combinable, but overall sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so description must compensate. It names three filter fields (category, intent, keyword) aligning with schema properties, but does not explicitly describe the limit parameter or its default behavior. Provides basic parameter context but misses full synergy.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool finds example records matching criteria (category, intent, keyword). It distinguishes itself from sibling tool count_records by specifying when to use each: 'use this for show me N examples' and 'prefer count_records for counts'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly provides when to use this tool ('show me N examples...') and when to use an alternative ('prefer count_records'). No ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• count_recordsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It discloses return behavior (no rows, only count/percentage) and safety (cheap, safe), but lacks details on permissions or side effects. Overall, it provides sufficient transparency for a read-only query tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: the first states purpose, the second provides usage context and safety. No unnecessary words, front-loaded with the core function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema (implied), the description does not need to detail return structure. It covers all three optional parameters, gives a real-world chain scenario, and mentions performance characteristics. The sibling tools context is well integrated.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It explains the three parameters (category, intent, keyword) as filters and gives an example with 'intent'. This adds meaning beyond the schema's names, though format constraints are not detailed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool counts records matching filters (category, intent, keyword), returning a number and percentage. It distinguishes from sibling tools like filter_records (which returns rows) and summarize_category, making the purpose specific and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides an explicit example ('how many refund requests?') and explains it is the counting half of a chain, implying when to use it versus filter_records. Also notes it is cheap and safe for large matches, guiding appropriate usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.