Server Quality Checklist

Tool Scores

• compare_datasetsA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It explains what the tool reports but does not disclose limitations, performance characteristics, or side effects (e.g., whether it loads entire files into memory).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences: purpose, output details, use cases. It is front-loaded with the key verb and resource, and every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description explains the return values in reasonable detail (deltas, column changes, etc.). However, it does not mention how max_rows affects the results or the output format (e.g., a structured table).

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage. The description implies path_a and path_b are the files to diff but does not explain the max_rows parameter or provide details on expected formats or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Diff two tabular files' and lists specific output metrics (row-count delta, column changes, dtype changes, null rates, means). This distinguishes it from sibling tools like column_stats or preview_data which focus on single datasets.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly provides use cases: comparing snapshots, validating data pipeline baselines, checking transformations. While it doesn't mention when not to use it or name alternatives, the intended contexts are clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• detect_quality_issuesA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations, but the description details the types of issues detected and the output structure (column, issue code, severity, explanation). It does not cover performance or modification (non-destructive implied), but it is transparent about the behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, front-loaded with the summary, then lists issues, then usage guidance. Every sentence adds value with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (many issue types) and lack of output schema, the description adequately covers the purpose, input, and output format. It is missing parameter details, but overall it is fairly complete for the tool's scope.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, meaning the description does not explain the parameters (path, max_rows). The user cannot infer what 'path' refers to or how 'max_rows' affects the audit. This is a critical gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Run a focused data-quality audit') and resource ('return issues grouped by severity'), and lists specific checks (duplicate rows, missing columns, etc.), clearly distinguishing it from sibling tools like profile_dataset or preview_data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit use cases: 'when the user cares specifically about cleanliness, is preparing data for modeling, or asks 'is anything wrong with this data?'. It does not explicitly mention when not to use or alternatives, but the guidance is clear and helpful.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• column_statsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It thoroughly explains the output for numeric, datetime, and text columns, and discloses error behavior. As a read-only analysis tool, no destructive actions are implied, but it could explicitly state that no modifications are made.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two well-structured paragraphs. The first sentence front-loads the purpose, followed by detailed bullet points. Every sentence adds value, and there is no redundancy or wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite lacking an output schema, the description provides a comprehensive overview of the returned statistics for each column type. It also covers error behavior. For a focused single-column analysis tool, this level of detail is complete and sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for parameter meanings. It only mentions the 'column' parameter in error context but fails to explain 'path' and 'max_rows'. This is a significant gap for a tool with 3 parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as a 'deep statistical dive on a single column' and details the statistics for different column types. It distinguishes from sibling tool 'profile_dataset' by suggesting usage after it for closer inspection, providing specificity and differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says to use 'after profile_dataset when one column needs closer inspection' and mentions error behavior when column is not found. While clear context is provided, there is no explicit exclusion of other sibling tools or when-not usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• suggest_dtypesA
  4.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Details specific behaviors: proposing numeric conversions, category for low-cardinality, downcasting, and reporting savings. Covers key expectations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three succinct sentences with no redundancy. Front-loaded with purpose, then behavior, then usage.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers purpose, usage, and behavior well for a simple tool. Minor gap: no parameter explanations, but overall adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema coverage, description should parameter details. It does not explain 'path' or 'max_rows', leaving their roles ambiguous.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool recommends memory-efficient or correct column dtypes, distinguishing it from sibling tools that handle stats, quality, or profiling.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit use cases: shrink memory footprint or fix wrong types. No exclusions or alternatives mentioned, but context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• preview_dataA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description takes on full burden, disclosing n capped at 100, mode options (head/tail/sample), and purpose. No side effects mentioned but likely none; adequate transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise, front-loaded sentences with no fluff; every word adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 3 parameters, no output schema, the description covers usage, parameters, and context adequately. Could mention return format explicitly but already covered.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% but description explains n (number of rows, capped at 100) and mode (with defaults and options), adding meaning beyond raw schema types. Path is implicit but clear.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool peeks at actual rows of a data file, distinguishing it from siblings like column_stats or profile_dataset by emphasizing real values instead of statistics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explains when to use (to see real example values for formatting, encodings, etc.) but does not explicitly state when not to use or list alternatives, though context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• profile_datasetA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  In the absence of annotations, the description fully carries the burden. It discloses that it reads the file, detects format, returns a structured overview, and describes the sampling behavior controlled by max_rows. It does not mention potential side effects (likely none) or performance limitations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a lead sentence, then bullet-like list. It is appropriately sized for the tool's complexity, though could be slightly more concise by merging some bullet points.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema, the description enumerates all major return categories: file metadata, shape, memory, missing values, duplicate count, per-column summary, and data-quality flags. For a two-parameter tool, this is fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description explains path's role as 'Reads the file at path' and max_rows adds meaning: 'caps how many rows are read (default: up to one million); pass 0 to remove the cap entirely.' This adds value beyond the schema's type and title.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Profile a tabular data file' and lists specific outputs (file metadata, shape, memory, missing values, column summary, flags). It distinguishes from sibling tools like preview_data and column_stats by emphasizing it's the fastest way to understand a dataset in one call.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly recommends 'Use this first whenever a user points you at a data file and wants to know what is in it.' This provides clear when-to-use guidance, though it doesn't explicitly mention alternatives or when not to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.