Server Quality Checklist

Tool Scores

• cancel_queryC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It states the action ('Cancel') but doesn't explain what cancellation entails (e.g., whether it's reversible, if it requires specific permissions, potential side effects, or rate limits). This leaves significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately brief and front-loaded with the core purpose. The Args/Returns sections are structured but slightly verbose for a single parameter; every sentence earns its place, though minor trimming could improve efficiency.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a cancellation operation (a mutation with potential side effects), no annotations, no output schema, and low schema coverage, the description is incomplete. It lacks details on behavioral traits, return value specifics, error conditions, and usage context, making it inadequate for safe and effective tool invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds minimal semantic context for the single parameter ('ID of the query to cancel'), but with 0% schema description coverage, it doesn't fully compensate. It clarifies that 'query_id' refers to a query ID, which is helpful, but doesn't provide format examples or validation rules beyond what the basic schema indicates.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Cancel') and target ('a running query'), providing a specific verb+resource combination. However, it doesn't differentiate from sibling tools like 'execute_query' or 'inspect_table', which would require explicit comparison to achieve a score of 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., that the query must be running), exclusions, or comparisons to sibling tools, leaving the agent without contextual usage instructions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• inspect_tableC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool retrieves metadata, implying a read-only operation, but doesn't specify permissions required, rate limits, error handling, or what 'detailed metadata' includes beyond a vague mention of 'columns, statistics, etc.' This leaves significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and concise, using a clear purpose statement followed by Args and Returns sections. Every sentence serves a purpose without unnecessary elaboration, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a metadata inspection tool with no annotations, no output schema, and low schema coverage, the description is incomplete. It lacks details on return format specifics, error cases, or behavioral traits, which are crucial for an agent to use the tool effectively in a database context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description lists the parameters (catalog, schema, table) and their roles, but with 0% schema description coverage, it only provides basic names without deeper context like format examples or constraints. It adds some value by clarifying what each parameter represents, but doesn't fully compensate for the lack of schema details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Get') and resource ('detailed metadata about a table'), making it easy to understand what the tool does. However, it doesn't explicitly differentiate from sibling tools like 'cancel_query' or 'execute_query', which might also interact with tables in different ways.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention any prerequisites, context for usage, or comparisons with sibling tools like 'cancel_query' or 'execute_query', leaving the agent to infer usage scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• execute_queryB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the action ('Execute a SQL query') and return type ('Dict[str, Any]: Query results including metadata'), but lacks critical details like permissions required, whether queries are read-only or mutating, timeouts, error handling, or rate limits. For a tool with no annotations, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized. It starts with a clear purpose statement, followed by bullet-like sections for 'Args' and 'Returns'. Each sentence adds value without redundancy, though the 'Returns' section could be more specific given the lack of an output schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (executing SQL queries), lack of annotations, and no output schema, the description is moderately complete. It covers the basic purpose and parameters but misses behavioral context (e.g., safety, performance) and detailed return value explanation. It's adequate as a starting point but has clear gaps for effective agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds significant value beyond the input schema, which has 0% description coverage. It explains each parameter's purpose: 'sql: The SQL query to execute', 'catalog: Optional catalog name to use for the query', and 'schema: Optional schema name to use for the query'. This clarifies semantics that the schema alone doesn't provide, though it doesn't detail format constraints or examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Execute a SQL query against Trino.' This specifies the verb ('Execute') and resource ('SQL query'), and identifies the target system ('Trino'). However, it doesn't explicitly differentiate from sibling tools like 'cancel_query' or 'inspect_table' beyond the core action.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools ('cancel_query', 'inspect_table'), suggest prerequisites, or outline scenarios where this tool is appropriate versus others. Usage is implied only by the tool's name and purpose.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.