Server Quality Checklist

Tool Scores

• get_files_listB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full responsibility for behavioral disclosure. It only states that the tool retrieves a list of files, implying a read operation, but gives no details about potential issues like file exclusions, access controls, or response format. This minimal disclosure is insufficient for a tool with zero annotation support.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with no extraneous words. It is front-loaded with the core action. However, it lacks structure or additional context that could be included without significant bloat, hence not a 5.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (no parameters, no output schema), the description is minimally adequate but incomplete. It fails to clarify what 'source of data' means or how the list is returned. For a straightforward listing tool, it meets the bare minimum but leaves gaps in understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has no parameters, and schema description coverage is 100% (empty). The description adds no parameter information, but since there are no parameters, the baseline score applies. The description does not need to elaborate further on parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource ('Get list of files that are source of data'), which clearly indicates the tool's purpose. However, the phrase 'source of data' is somewhat ambiguous, lacking precision about what types of files or sources are included. It distinguishes itself from siblings like 'execute_polars_sql' and 'get_schema' by focusing on file listing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No usage guidance is provided. There is no explicit instruction on when to use this tool versus alternatives, nor any context about prerequisites or applicability. The description only states what the tool does, leaving the agent to infer usage from the name and sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_schemaB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, and the description does not disclose any behavioral traits beyond the basic action. For instance, it doesn't mention what happens if the file is missing or the schema is unavailable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no superfluous words. It is front-loaded and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simplicity of the tool (2 params, no output schema), the description is adequate but minimal. It doesn't cover error conditions or return details, though the core functionality is clear.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds no meaning beyond the input schema for file_location, and file_type already has a description in the schema. With 50% coverage, the description does not compensate for the missing param info.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool gets the schema of a single data file, differentiating it from sibling tools like execute_polars_sql and get_files_list. However, it could be more specific about the schema format or structure returned.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives, nor any prerequisites or context for usage. The description is purely functional without usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• execute_polars_sqlB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears full responsibility for behavioral disclosure. It mentions reading files and executing SQL, but does not detail side effects (likely none), performance implications, error handling (e.g., incompatible schemas), or return value structure. This is insufficient for a data-execution tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences long, immediately stating the core purpose and key constraints. No redundant or extraneous information, and the structure is clear and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is moderately complex (file reading, SQL execution), yet the description omits details on return format, error behavior, supported file paths, and limitations. Without an output schema, the agent is left without enough information to fully understand the tool's behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 67%, and the description adds value for 'file_locations' (must have same schema) and 'query' (must use 'self'). However, the extensive list of supported functions is already in the input schema for 'query', so the description only marginally supplements with constraint context. Baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool reads data from file locations and executes a Polars SQL query, returning the result. It specifies the action (reads, executes), resource (file data, SQL query), and output. This differentiates it from sibling tools (get_files_list, get_schema) which focus on listing files and schemas respectively.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description notes key constraints: all files must share the same schema and columns, and the SQL query must refer to the source data as 'self'. However, it does not explicitly state when to use this tool over alternatives or when not to use it, leaving the agent to infer from context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.