Server Quality Checklist

Tool Scores

• list_filesA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden of behavioral disclosure. It explains the filtering behavior (data_files_only limiting to DuckDB-readable extensions) and the recursive option, adding meaningful context beyond the bare 'list files'. However, it does not disclose edge cases like error handling or hidden files, preventing a perfect score.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single purpose sentence followed by a concise bulleted list of arguments. Every sentence is informative, the structure is clear and front-loaded, and there is no unnecessary verbosity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given an output schema exists, the description doesn't need to explain return values. It covers the tool's purpose, parameters, and filtering behavior comprehensively for a simple listing tool. It could be slightly more explicit about how this fits into the overall workflow, but it is largely complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has no parameter descriptions (0% coverage), so the description fully compensates by explaining each parameter's meaning and providing an example glob pattern. This adds significant value beyond the schema's type/default information.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists data files in a local directory or S3 prefix, using the specific verb 'list' and resource 'data files', which immediately distinguishes it from siblings like query, describe_file, preview_file, and profile_columns.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The purpose is evident, but there is no explicit guidance on when to use this tool versus alternatives like describe_file or preview_file. Usage is implied from the description ('list data files'), but no exclusions or alternative scenarios are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• preview_fileA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosing behavior. It states the tool shows the first rows and supports glob/URL paths, which implies read-only behavior. However, it does not mention error handling, file size limits, or output format details. It is adequate but minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exceptionally concise with two sentences, front-loaded with the primary purpose. The parameter list is minimal and each element adds value, with no redundant or extraneous information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Since an output schema exists, return values are already covered. The description successfully conveys the core purpose and parameters, and the tool is simple. However, it lacks explicit guidance on when to prefer this over sibling tools (e.g., describe_file, profile_columns), leaving a small gap in contextual guidance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema property descriptions are 0% covered, and the description fully compensates: 'path' is explained as 'File path, glob or URL' and 'rows' as 'How many rows to show (default 20)'. This adds meaningful semantics beyond the bare types and default value in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'show' with the resource 'data file' and scope 'first rows', clearly distinguishing it from siblings like describe_file (schema/metadata) and profile_columns (statistics). The phrase 'see real values' reinforces the data-inspection purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'so you can see real values' implies a clear use case for inspecting actual data, and the tool's role is evident from its name and siblings. However, it does not explicitly state when not to use it or name alternatives, so it earns a 4 rather than a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• describe_fileA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the behavioral transparency burden. It discloses the row count behavior (can be skipped for large inputs) and the supported path types. It does not explicitly state side effects, but as a read-only describe operation, the behavior is sufficiently clear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured: a clear opening statement, a brief list of supported formats, and an Args section. No unnecessary words or redundancy, and every sentence serves a purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool only has two parameters (one required) and an output schema exists, the description is complete. It covers the purpose, supported input types, parameter semantics, and the optional row count behavior. The output schema covers return values, so no further detail is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The Args section explicitly describes both parameters: path with an example ('data/sales_*.parquet') and include_row_count with its purpose. This fully compensates for the 0% schema description coverage and adds valuable meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Return the column names and types of a data file, plus its row count.' This uses a specific verb ('return') and resource ('data file') and distinguishes it from siblings like query, preview_file, and profile_columns, which have different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context on applicable inputs: 'Works for any path DuckDB can read -- csv, parquet, json/ndjson, xlsx, compressed variants, globs matching many files, http(s) URLs and s3 URIs.' It also explains the include_row_count option for large inputs. However, it does not explicitly mention when to use this tool instead of siblings like preview_file or profile_columns.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• profile_columnsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the tool scans the whole file (cost), that distinct counts are approximate, and that most-frequent values are only for low-cardinality columns. These are useful behavioral traits. It does not discuss error conditions or permissions, but these are less critical for a read-only profiling tool with an output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences plus a clear bulleted Args list. It front-loads the purpose, gives one usage guideline, and then details parameters. Every sentence adds value with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema, return values are covered. The description covers purpose, usage, parameters, and performance (whole-file scan). It is slightly incomplete in not addressing error handling or clarifying the exact output structure, but the output schema mitigates this. Overall it is sufficiently complete for a profiling tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, but the description's Args section provides meaning for all three parameters: path (file path/glob/URL), columns (restriction, default all), and top_k (number of values, 0 to skip). This fully compensates for the schema's lack of descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Profile a file's columns' which is a specific verb and resource, and lists concrete outputs (null counts, approximate distinct counts, min/max, most frequent values). It also differentiates from the sibling tool describe_file by explicitly saying 'prefer describe_file when you only need types.'

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear when-to-use guidance: it states this tool 'scans the whole file' and directs users to 'prefer describe_file when you only need types.' This gives an explicit alternative and context for when this tool is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• queryA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full behavioral disclosure. It states the read-only nature, accepted and rejected statement types, the markdown output format, and the max_rows default. This goes beyond minimal requirements and helps the agent understand important constraints and return format.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded with the purpose, followed by necessary constraints and an example. It includes an Args section that cleanly maps to parameters without unnecessary verbosity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a two-parameter tool with no annotations, the description covers purpose, accepted/rejected inputs, output format, and parameter semantics. It also clarifies the read-only nature and file-reading capability, making it self-sufficient for an agent to use correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains sql as 'the statement to run' and max_rows as a 'row cap' with a default to the server's limit, plus includes an example for reading files. This adds significant meaning beyond the raw schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs a read-only DuckDB SQL statement and returns rows as a markdown table. It specifies the verb 'Run' and resource 'DuckDB SQL statement', and distinguishes itself from file-oriented siblings (describe_file, preview_file, etc.) by focusing on arbitrary SQL execution.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides clear context: accepted statement types (SELECT, WITH, DESCRIBE, SUMMARIZE, SHOW, EXPLAIN) and explicitly rejects writes. However, it does not mention when to prefer this tool over sibling tools or provide exclusions beyond statement type, so it lacks explicit alternative guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.