Server Quality Checklist

Tool Scores

• describe_tableA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the readOnlyHint/idempotentHint annotations, the description discloses the exact return format (Markdown with row count, columns, semantic definitions, join keys, caveats) and error behavior, including the error message with available tables. This is rich behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (main description, Args, Returns, Examples). Every sentence serves a purpose, and it is front-loaded with the core functionality. Length is justified by the depth of useful guidance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, read-only, with output schema), the description is complete: it explains what it does, when to use it, what it returns, and error handling. No critical information is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description already covers the parameter ('Table name, e.g. 'orders'. Case-insensitive.'). The description's Args section repeats the same information without adding new meaning, so baseline 3 for high schema coverage is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource: 'Describe one table: columns, types, null rates, distinct counts, examples.' It clearly differentiates from siblings by noting when not to use it and recommending profile_column for single-column distribution.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance is provided: 'Call this before writing a query against an unfamiliar table' with concrete use cases and a non-use case that names an alternative tool (profile_column). This clearly explains when to use this tool versus siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explainA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond annotations (readOnlyHint=true, idempotentHint=true), the description explains that estimates come from DuckDB's optimizer and are not guarantees. It also discloses the failure guard for non-SELECT input and the markdown return format, adding meaningful behavioral context beyond what annotations provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Every sentence serves a purpose: purpose, usage guidance, parameter note, return value, failure handling, and examples. The sectioned format with bolded examples is well-structured and front-loaded with the key point (no execution). No fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a near-complete specification: purpose, usage, return format, failure behavior, and guard conditions. The single parameter is fully documented in the schema, and the description clearly explains the output type (Markdown string) even without the output schema being shown.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already describes the sql parameter in great detail (100% coverage), including supported constructs and rejected statements. The 'Args' line in the description restates what the schema says without adding new semantics. Baseline 3 is appropriate since schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Show') and resource ('query plan') and immediately clarifies the tool does not execute the query. This cleanly distinguishes it from sibling tools like 'query' and other data operations. The scope is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly states when to use the tool (expensive queries, timeouts) and when not to use it ('you just want the answer (call query directly)'). It names the alternative sibling tool directly, fully satisfying the when/when-not/alternatives criterion.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• profile_columnA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false. The description adds substantial behavioral context beyond these: it explains that the profile adapts to column type (numeric/date/text) with specific statistics and warnings, describes the return format as Markdown, and even includes an error message example for invalid input. This goes far beyond annotation data.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (summary, type-specific behavior, Args, Returns, Examples), uses bullet lists for readability, and front-loads the core purpose in the first line. Every sentence earns its place, providing rich detail without fluff or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with only two parameters, the description is thorough: it covers all column type behaviors, return format, error handling, and typical use cases. It also leverages an output schema (though not shown) and annotations, making it fully contextual for an agent to select and invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides 100% description coverage for both parameters (table and column), including examples and case-insensitivity. The description's Args section essentially repeats this information without adding new meaning. Baseline is 3 due to high schema coverage; the description adds no extra parameter semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Profile one column: distribution, outliers, coverage gaps, dirty values', which is a specific verb+resource combination that clearly states what the tool does. It further distinguishes itself from sibling tools like describe_table by explicitly saying 'Don't use when: you want the whole schema (use describe_table)' and by focusing on single-column profiling.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit 'Use when' scenarios with concrete example questions ('Is unit_price skewed?', 'Are there missing days in order_date?') and a non-example ('a GROUP BY returned more groups than expected'). It also gives a clear exclusion ('Don't use when: you want the whole schema') and names the alternative (describe_table).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_datasetsA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already mark this as read-only/idempotent, and the description adds valuable behavioral context: the output is deliberately small and cheap, it returns a Markdown table plus caveats, and it notes failure returns 'Error: <message>'. This goes well beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise yet complete: a clear one-sentence summary, a 'Start here' hint, a structured Returns section, and use/non-use examples. Every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter discovery tool, the description covers purpose, when to use, output format, failure behavior, and even the caveats section. It is fully sufficient for an agent to invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so there are no parameter semantics to explain. Per the rubric, a baseline of 4 applies; the description correctly omits parameter details since none exist.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'List every table in the warehouse with its row count, grain, and caveats.' It clearly distinguishes from siblings by contrasting with describe_table ('Don't use when... need its columns').

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit when-to-use guidance: 'Start here' and examples for use ('What data do I have access to?') and non-use ('use describe_table instead'). It names the alternative tool, making the decision clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• queryA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the readOnlyHint and idempotentHint annotations, the description discloses important enforcement behaviors: non-SELECT statements are rejected, LIMIT is injected or clamped, time limits cancel execution, and truncation is explicitly reported with total_counts. This gives the agent a detailed model of what will happen and why.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Although long, the description is dense and well-structured, with sections for enforcement, usage guidance, arguments, return values, examples, and error handling. Every section adds practical information and the key purpose and safety characteristics are front-loaded in the first sentence.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description fully covers the tool's complexity: it explains safety enforcement, truncation semantics, error recovery paths, concrete examples for appropriate use, and output formats. The presence of an output schema does not reduce the need for this behavioral context, and the description delivers it comprehensively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already has 100% parameter coverage with detailed descriptions, so the baseline is 3. The description adds extra meaning by explaining the enforcement behavior of max_rows (clamped and lowered), the token-cost tradeoff of response_format, and the concrete return shapes associated with each format, going beyond the schema's field-level definitions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The opening sentence clearly states the tool runs one read-only SELECT against the warehouse and returns rows, specifying both the verb and resource. It distinguishes itself from sibling metadata tools like describe_table and list_datasets by being the SQL query tool for actual data retrieval.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit when-to-use and when-not-to-use guidance, including direct examples like aggregating in SQL versus selecting raw rows and calling describe_table first when column names are unknown. It also says what not to use the tool for, making it easy for an agent to choose among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.