Server Quality Checklist

Tool Scores

• summaryC
  2.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must fully disclose behavioral traits. It does not mention whether the tool modifies data (likely read-only, but unstated), if it works on non-numeric columns or raises errors, or if there are limits on dataset size. The return format is not described, though an output schema exists; however, the schema is not shown and the description adds no behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short and to the point, with a simple one-line summary and a single parameter in an Args block. However, the 'whole dataset' claim is ambiguous (summary of all columns? or just one column?). The structure is acceptable but the content is too sparse to be effective.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 1 parameter, no annotations, no enum constraints, and an output schema (though not detailed). Given the simplicity, the description could be more complete. It doesn't explain what output to expect, how to interpret the results, or error handling. For a numeric analysis tool, this is insufficient for correct agent invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for the schema's lack of parameter documentation. It only says 'name: the dataset name', which is minimal and adds no extra semantics—no format, no examples, no clarification of case sensitivity or allowed characters. A baseline of 3 is not merited because the description fails to add value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool returns 'summary statistics' for a column or whole dataset, but 'summary statistics' is vague—it doesn't specify what statistics (e.g., mean, median, count, missing values). The sibling tools include load_csv, filter_rows, top_rows, and correlation, but the description fails to differentiate summary from correlation or top_rows, which also provide data insights.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given on when to use this tool versus siblings. For example, it doesn't clarify when to use summary versus correlation for numerical analysis, or top_rows for previewing data. There's no mention of prerequisites (e.g., dataset must be loaded via load_csv), or when this is preferred over other analytical tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• top_rowsA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden for behavioral disclosure. It states sorting is descending, which is good, but doesn't mention whether ties are handled, if the tool modifies the dataset (likely no, but not stated), permissions needed, performance implications for large datasets, or what happens if the column doesn't exist. For a data query tool, the lack of safety/read-only indication is a gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: one sentence for the purpose, then a bullet-like Args section. Every line adds value, no filler or redundancy. It's also front-loaded with the main action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (context signal), the description doesn't need to explain return values. It is relatively complete for a simple query tool: states the sorting direction, documents all params, and provides a default. The only missing context is behavioral specifics (e.g., read-only, error handling) and tie-breaking, but these are secondary for a straightforward sorted row retrieval.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It does so by clearly documenting each parameter: 'name: the dataset name', 'column: the column to sort by', 'n: how many rows to return (default 5).' This adds semantic meaning beyond the schema's minimal 'Name' and 'Column' titles, including the default value for 'n'.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Return the top-N rows sorted by column descending.' This uses a specific verb (Return) and resource (top-N rows) with sorting direction clarified, which distinguishes it from siblings like 'filter_rows' (which filters, not sorts) and 'summary' (which aggregates). However, it doesn't explicitly contrast itself against these siblings in the description.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage via the Args section, indicating when each parameter is needed. However, it provides no explicit guidance on when to use this tool vs alternatives like 'filter_rows' for row selection or 'summary' for statistical overviews. The sibling tools suggest a data exploration workflow, so a brief note on context would help.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• correlationA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the main action (return Pearson correlation) but does not disclose whether the operation is read-only or destructive, error handling for missing or non-numeric columns, or any side effects. For a tool with no annotations, this is insufficient transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description consists of one clear sentence followed by a minimal, structured parameter list. Every part is essential: the verb, the specific correlation type, and the parameter explanations. No unnecessary words or repetition. The most important information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (a statistical function) and the presence of an output schema (not shown but indicated), the description covers the core functionality and parameter semantics adequately. It does not describe error conditions or assumptions (e.g., columns must be numeric), but for a correlation tool, the provided information is sufficient for correct invocation in most cases.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema properties have no descriptions (0% coverage). The description adds meaning by explaining each parameter: 'name: the dataset name', 'col_a: first numeric column', 'col_b: second numeric column'. This clarifies the purpose of each parameter beyond the schema titles, though it could be more precise (e.g., dataset name format).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states 'Return the Pearson correlation between two numeric columns' which is a specific verb (Return) and resource (Pearson correlation). This clearly distinguishes it from sibling tools like load_csv, list_datasets, summary, filter_rows, and top_rows, which perform different operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide explicit guidance on when to use this tool versus alternatives. While the mention of 'Pearson correlation' implies it is for measuring linear relationships between two numeric columns, no when-not-to-use instructions or alternative tool mentions are given. The context of sibling tools partially clarifies its distinct role.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• filter_rowsA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It does not specify whether the filter modifies the dataset in-place or returns a new filtered view, nor does it mention side effects, permissions, or performance implications. The description is minimal and lacks behavioral disclosure beyond the basic operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise: one line explaining the core action plus a bullet list of arguments. Every sentence earns its place; there is no fluff. The structure is logical and easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has an output schema (not shown), so the description does not need to detail return values. All four required parameters are explained. Minor omissions: no mention of error conditions or what happens if the column is not numeric. Overall, it is reasonably complete for a simple filter tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, but the tool description provides clear, human-readable explanations for all four parameters: name, column, operator (with list of valid operators), and value. This adds significant meaning beyond the schema's bare titles, compensating for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool filters rows using a column, operator, and value, with a concrete example (spend > 5000). It distinguishes from sibling tools like summary, top_rows, and correlation, which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for filtering numeric columns with comparison operators, but provides no explicit guidance on when to use this tool versus alternatives, nor any prerequisites or exclusions. The example helps, but it is not a full usage guideline.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• load_csvA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses that the tool registers a dataset and returns the inferred schema, and mentions CSV format (first row is header). However, it omits important behavioral traits: what happens if the name already exists (overwrite/error), whether the operation is reversible, or any size/performance limits. This is adequate but has gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: two sentences plus a parameter list. Every sentence is substantive, with no filler. The purpose is stated first, followed by parameter details, which is an ideal structure.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 required params, no nested objects, output schema present), the description covers the core functionality and parameter roles. However, it lacks details on duplicate name handling, error scenarios, or data format constraints (e.g., encoding). This is a minor gap for a load operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It fully explains both parameters: 'name' is a unique name, 'csv_text' is raw CSV content with comma-separation and first-row header. This adds significant meaning beyond the schema's type/title alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action: 'Register a dataset from CSV text.' This is a specific verb+resource, and it distinguishes itself from sibling tools like list_datasets, filter_rows, and correlation, which are for querying or analyzing existing data, not loading.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool vs alternatives. It does not mention prerequisites (e.g., ensure CSV is valid) or when not to use it (e.g., if the dataset already exists). Sibling tools are not referenced for context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_datasetsA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. 'List' implies a read-only operation with no side effects, which is sufficient. The description also notes that schemas are included, providing basic behavioral context. However, no details about performance, ordering, or caching are given.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that is perfectly concise and front-loaded. Every word adds value, and there is no unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters, the presence of an output schema, and recognizable sibling tools, the description is largely complete. It explains what the tool does and what it returns ('with their schema'). A minor improvement could mention ordering or pagination, but it is not necessary for a simple list operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and schema coverage is 100% (trivially). The description adds no parameter information because none exist, which is appropriate. According to guidelines, 0 parameters yields a baseline of 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'List all registered datasets with their schema' uses a specific verb ('List') and resource ('registered datasets'), clearly distinguishing it from sibling tools like load_csv, summary, and filter_rows which operate on specific datasets.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly states the tool's function as a listing operation, implying it should be used as a starting point before selecting or manipulating datasets. While no explicit exclusions or alternatives are given, the context from sibling tools makes the usage straightforward.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.