Server Quality Checklist

Tool Scores

• steampipe_table_listB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions that the tool 'lists' tables, which implies a read-only operation, but doesn't specify whether this requires authentication, how results are returned (e.g., pagination, format), or any rate limits. The description adds minimal behavioral context beyond the basic action.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise with just two sentences that are front-loaded and waste-free. The first sentence states the core purpose, and the second adds essential usage guidance, making every word earn its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (2 optional parameters, no output schema, no annotations), the description is minimally adequate. It covers the basic purpose and hints at parameter usage but lacks details on behavioral aspects like authentication, result format, or error handling, which would be helpful for an agent to use it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description mentions that 'schema and filter parameters' can be used to 'narrow down results,' which adds some context about their purpose. However, with 100% schema description coverage, the input schema already fully documents both parameters, including their types, optionality, and usage details (e.g., ILIKE syntax for filter). The description provides only marginal value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List all available Steampipe tables') and the resource ('Steampipe tables'), making the purpose immediately understandable. However, it doesn't explicitly differentiate this tool from its sibling 'steampipe_table_show' which likely shows details of a specific table rather than listing all tables.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides implied usage guidance by mentioning that schema and filter parameters can be used to 'narrow down results,' suggesting this tool is for listing tables with optional filtering. However, it doesn't explicitly state when to use this tool versus alternatives like 'steampipe_table_show' or 'steampipe_query,' nor does it provide exclusion criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• steampipe_table_showB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It describes the tool's function but lacks details on behavioral traits such as error handling (e.g., what happens if the table doesn't exist), performance characteristics, or output format. This is a significant gap for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that efficiently conveys the tool's purpose and scope without unnecessary words. It's front-loaded with the main action and resource, making it easy to understand at a glance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, no output schema, no annotations), the description is minimally adequate. It covers the purpose but lacks behavioral context and output details, which are important for a tool that retrieves metadata. Without annotations or output schema, more completeness would be beneficial.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents both parameters ('name' and 'schema') with clear descriptions. The description doesn't add any parameter-specific information beyond what's in the schema, such as examples or constraints. Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get detailed information') and resource ('specific Steampipe table'), and specifies the scope of information returned ('including column definitions, data types, and descriptions'). It distinguishes from the sibling 'steampipe_table_list' by focusing on details for a single table rather than listing tables. However, it doesn't explicitly contrast with 'steampipe_query' which might also return table information through queries.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when detailed metadata about a specific table is needed, but doesn't provide explicit guidance on when to use this tool versus alternatives like 'steampipe_table_list' for listing tables or 'steampipe_query' for querying data. It mentions the resource type ('Steampipe table') but lacks context on prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• steampipe_plugin_showA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions what details are returned (version, memory limits, configuration) but does not cover other important aspects such as error handling (e.g., what happens if the plugin doesn't exist), permissions required, or whether it's a read-only operation. This leaves significant gaps for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that efficiently conveys the tool's purpose and key details. It is front-loaded with the main action and resource, and every part of the sentence adds value without redundancy or unnecessary information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (simple read operation with one parameter) and the lack of annotations and output schema, the description is moderately complete. It specifies what details are retrieved, but it does not fully compensate for the missing behavioral context (e.g., error handling, permissions) or describe the return format, which would be needed for higher completeness in the absence of an output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, with the parameter 'name' clearly documented in the schema as 'Name of the plugin to show details for.' The description does not add any additional meaning or context beyond what the schema provides, such as format examples or constraints. With high schema coverage, the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Get details') and resource ('a specific Steampipe plugin installation'), distinguishing it from siblings like 'steampipe_plugin_list' (which lists all plugins) and 'steampipe_table_show' (which shows table details). It explicitly mentions what details are included: version, memory limits, and configuration.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by specifying 'a specific Steampipe plugin installation,' suggesting it should be used when details for a particular plugin are needed. However, it does not explicitly state when not to use it or name alternatives like 'steampipe_plugin_list' for listing all plugins, which would be required for a score of 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• steampipe_plugin_listA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It describes what the tool does (lists plugins) and provides useful context about plugins providing access to data sources, but doesn't disclose behavioral traits like whether this requires specific permissions, how results are formatted, or if there are any rate limits. It adds some value but lacks operational details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with zero waste. First sentence states the core purpose, second adds helpful context about plugins. Well-structured and front-loaded with the essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, no output schema, and moderate complexity (listing operation), the description is adequate but incomplete. It explains what the tool does and provides context about plugins, but lacks details about return format, error conditions, or operational constraints that would be helpful for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters with 100% schema description coverage, so no parameter documentation is needed. The description appropriately doesn't discuss parameters, maintaining focus on the tool's purpose. Baseline for 0 parameters is 4, as it avoids unnecessary parameter discussion.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('List all') and resource ('Steampipe plugins installed on the system'), with additional context about what plugins provide. It distinguishes from siblings like steampipe_plugin_show (detail view) and steampipe_table_list (different resource type).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when needing to see installed plugins, but doesn't explicitly state when to use this vs. alternatives like steampipe_plugin_show for detailed plugin information or steampipe_table_list for table listings. No explicit exclusions or prerequisites are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• steampipe_queryA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key traits: queries are 'read-only,' must use 'PostgreSQL syntax,' and includes performance tips (limit columns, use CTEs, trust search path). This covers safety, syntax, and optimization, though it lacks details on error handling or output format.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded, starting with the core purpose. Each sentence adds value: purpose, constraints, performance tips, and sibling tool references. There is no wasted text, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (querying multiple data sources) and lack of annotations/output schema, the description does a good job covering key aspects: purpose, constraints, performance, and sibling tool integration. However, it doesn't explain return values or error cases, leaving some gaps for a tool with no output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, with the parameter 'sql' fully documented in the schema. The description adds minimal semantics beyond this, only reiterating that SQL must be 'read-only' and use 'PostgreSQL syntax,' which is already in the schema. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Query cloud infrastructure, SaaS, APIs, code and more with SQL.' It specifies the verb ('query') and resources, but doesn't explicitly differentiate from sibling tools like steampipe_table_list/show, which are mentioned for checking tables rather than querying. This makes it clear but not fully sibling-distinguished.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool vs alternatives: it instructs to 'Check available tables and columns before querying using steampipe_table_list and steampipe_table_show.' This clearly defines a usage sequence and distinguishes it from sibling tools, offering practical alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.