Server Quality Checklist

Tool Scores

• get_statusC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but only states it retrieves status without disclosing behavioral traits like permissions needed, rate limits, response format, or error handling. It lacks context on what 'status' entails (e.g., pending, completed, failed).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with zero wasted words, front-loading the purpose efficiently. It's appropriately sized for a simple tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description is incomplete for a tool that likely returns status details. It doesn't explain what 'status' includes or how to interpret results, leaving gaps for an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents the 'queryExecutionId' parameter. The description adds no additional meaning beyond implying it's used to fetch status, meeting the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get') and resource ('current status of a query execution'), making the purpose immediately understandable. However, it doesn't differentiate from sibling tools like 'get_result' or 'run_query', which might also relate to query execution status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'get_result' or 'run_query'. The description implies usage for checking status but doesn't specify prerequisites, timing, or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_saved_queryC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure but offers minimal information. It states the tool executes a query but doesn't cover critical aspects like whether it's read-only/destructive, authentication needs, rate limits, error handling, or what happens after execution (e.g., does it return results immediately or trigger an async process?). This leaves significant gaps for a tool that likely interacts with a database system.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without any fluff or redundancy. It's front-loaded with the core action and resource, making it easy to parse quickly. Every word earns its place in conveying essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of executing database queries (with 4 parameters, no annotations, and no output schema), the description is incomplete. It doesn't explain what the tool returns (results, status, or something else), how it handles errors, or its interaction with siblings like 'get_result' and 'get_status'. For a tool with potential side effects and no structured safety hints, more context is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all four parameters with their types, descriptions, and constraints. The description adds no parameter-specific information beyond what's in the schema, which meets the baseline for high schema coverage but doesn't provide extra value like explaining relationships between parameters or usage examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Execute') and resource ('a saved (named) Athena query by its query ID'), making the purpose immediately understandable. However, it doesn't differentiate this tool from sibling tools like 'run_query' or explain how saved queries differ from ad-hoc queries, which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'run_query' or 'list_saved_queries'. It doesn't mention prerequisites (e.g., needing a saved query ID from 'list_saved_queries') or typical use cases, leaving the agent with no contextual decision-making help.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_saved_queriesB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool lists queries, implying a read-only operation, but doesn't clarify if it requires specific permissions, has rate limits, returns paginated results, or what the output format looks like (e.g., JSON list of query names). For a tool with zero annotation coverage, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence that efficiently conveys the core purpose without any wasted words. It's front-loaded with the key action ('List all saved...'), making it easy to scan and understand. Every part of the sentence contributes directly to defining what the tool does, earning a perfect score for conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema, no annotations), the description is adequate as a basic overview. It covers the what and where, but lacks details on behavioral aspects like permissions, output format, or integration with siblings. For a read-only listing tool, this is minimally viable but leaves gaps that could hinder effective use by an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, meaning no parameters are documented in the schema. The description doesn't add parameter details, which is appropriate since there are none to explain. This earns a baseline score of 4, as the description doesn't need to compensate for missing schema information and avoids unnecessary complexity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List') and resource ('all saved (named) Athena queries'), making the purpose immediately understandable. It specifies the scope ('available in your AWS account'), which helps distinguish it from tools that might operate on different resources or scopes. However, it doesn't explicitly differentiate from sibling tools like 'run_saved_query' or 'get_result', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing AWS credentials), compare it to siblings like 'run_saved_query' for executing queries or 'get_result' for retrieving results, or indicate scenarios where listing queries is appropriate (e.g., before selecting one to run). This lack of contextual direction leaves the agent to infer usage from the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_resultA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adds value by specifying that it returns an error for running queries, which is a key behavioral trait. However, it lacks details on other aspects like rate limits, authentication needs, or what the results format looks like (e.g., structured data, pagination). This leaves gaps for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is highly concise and front-loaded, consisting of only two sentences that directly state the tool's purpose and a critical behavioral constraint. Every word earns its place, with no redundancy or unnecessary information, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, no output schema, no annotations), the description is partially complete. It covers the core purpose and a key error condition but lacks details on output format, error types, or integration with siblings. Without annotations or an output schema, more context on what 'results' entail would improve completeness for effective agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with clear documentation for both parameters: 'queryExecutionId' and 'maxRows' (including default and constraints). The description does not add any semantic details beyond what the schema provides, such as explaining what a query execution ID is or how maxRows affects performance. Thus, it meets the baseline but doesn't enhance parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get results for a completed query.' It specifies the verb ('Get') and resource ('results'), and distinguishes it from siblings like 'get_status' (which likely checks query status) and 'run_query' (which initiates queries). However, it doesn't explicitly differentiate from 'list_saved_queries' or 'run_saved_query,' keeping it from a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool: 'for a completed query.' It implies an alternative by stating 'Returns error if query is still running,' suggesting 'get_status' should be used first to check completion. However, it doesn't explicitly name alternatives or provide exclusions, such as when not to use it (e.g., for saved queries without execution).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_queryA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adds useful context beyond basic function, such as the timeout behavior (returns queryExecutionId if timeout occurs) and that it returns full results otherwise. However, it lacks details on permissions, rate limits, error handling, or what 'full results' entail, which are important for a mutation-like tool like query execution.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, consisting of two concise sentences that directly convey the tool's purpose and key behavioral trait (timeout handling). Every sentence earns its place by providing essential information without redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (executing SQL queries with potential timeouts), lack of annotations, and no output schema, the description is somewhat complete but has gaps. It covers the basic operation and timeout behavior but misses details on output format, error cases, or integration with siblings like get_result, which could aid the agent in proper usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description does not add any parameter-specific semantics beyond what the schema provides (e.g., it doesn't explain query syntax or database naming conventions). Baseline 3 is appropriate as the schema handles the heavy lifting, but the description doesn't compensate with extra insights.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Execute a SQL query using AWS Athena') and resource (SQL queries), distinguishing it from siblings like get_result (retrieves results), get_status (checks status), list_saved_queries (lists saved queries), and run_saved_query (executes saved queries). It precisely defines what this tool does versus alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool (to execute SQL queries with AWS Athena) and implies when not to use it (e.g., for retrieving results or checking status, which are handled by siblings). However, it does not explicitly name alternatives or state exclusions, such as preferring run_saved_query for saved queries, leaving some guidance implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.