Server Quality Checklist

Tool Scores

• dbt_runA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively explains what the tool does (executes dbt models to transform data), mentions the output format (text, not JSON), and implies it's a write operation that changes data in the warehouse. It could be more explicit about potential side effects like data overwriting or performance impact.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized with two paragraphs that efficiently convey purpose, usage context, and output format. The first sentence clearly states the core function, and the 'Returns' section provides important behavioral information. There's minimal wasted text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 6-parameter tool with no annotations but 100% schema coverage and an output schema, the description provides good context about what the tool does and when to use it. It explains the output format limitation (text not JSON) which is valuable. The main gap is lack of explicit guidance on when NOT to use it versus sibling tools like dbt_build or dbt_compile.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so all parameters are well-documented in the schema itself. The description doesn't add any parameter-specific information beyond what's already in the schema descriptions, so it meets the baseline expectation without providing additional value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('execute dbt models', 'transform data', 'build analytical tables') and distinguishes it from siblings by mentioning its role in refreshing data or implementing new transformations. It explicitly names the resource (dbt models) and the action (run).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool ('when it needs to execute dbt models to transform data and build analytical tables'), including essential use cases like refreshing data or implementing new transformations. However, it doesn't explicitly mention when NOT to use it or name specific alternatives among the sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• dbt_seedA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and does well by explaining the tool's purpose and use cases. It adds valuable context about the output format ('Returns: Output from the dbt seed command as text (this command does not support JSON output format)'), which is crucial behavioral information not in the input schema. However, it doesn't mention potential side effects like database writes or performance implications.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with purpose first, then usage guidelines, and finally output format. Every sentence adds value, though the output format section could be slightly more integrated rather than a separate 'Returns:' block.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Good completeness for a data loading tool with no annotations but an output schema. The description covers purpose, usage context, and output behavior. With 4 parameters fully documented in the schema and an output schema present, the main gap is lack of explicit warnings about database writes or permissions needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, providing good documentation for all 4 parameters. The description doesn't add any parameter-specific information beyond what's in the schema, so it meets the baseline of 3 where the schema does the heavy lifting without compensating with extra semantic context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Load CSV files as seed data') and resource ('database'), distinguishing it from siblings by focusing on initial data loading from CSV files rather than building, compiling, testing, or other dbt operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use this tool ('when it needs to load initial data from CSV files into the database') and provides concrete use cases ('creating reference tables, test datasets, or any static data that models will depend on'), giving clear context for selection among sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• dbt_buildA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses key behavioral traits: the command runs multiple components 'in the correct order' and returns 'output as text (this command does not support JSON output format).' However, it doesn't mention potential side effects like database mutations, execution time, or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with purpose first, usage guidelines second, and return format third. The three sentences each earn their place, though the second sentence could be slightly more concise by combining the two usage scenarios.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (multi-step build process) and 100% schema coverage with an output schema present, the description provides excellent contextual completeness. It explains the comprehensive nature of the operation, when to use it, and clarifies the output format limitation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all 6 parameters thoroughly. The description adds no parameter-specific information beyond what's in the schema, meeting the baseline expectation when schema coverage is high.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'Run build command (seeds, tests, snapshots, and models)' with specific verb+resource. It distinguishes from siblings by explaining this is a 'comprehensive build process' that runs multiple components in order, unlike single-purpose tools like dbt_seed or dbt_test.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use: 'when it needs to execute a comprehensive build process... ideal for complete project deployment or ensuring all components work together.' This provides clear context for choosing this tool over simpler alternatives like dbt_run or dbt_test.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• dbt_depsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes the tool's function (installing/updating dependencies) and output format (text from the dbt deps command, not JSON). However, it doesn't mention potential side effects like network dependencies, time requirements, or error conditions, which would be helpful for a mutation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured with two sentences: the first states the purpose and usage guidelines, and the second explains the output format. Every sentence adds value without redundancy, and the information is front-loaded with the core functionality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (installing dependencies), no annotations, and an output schema that exists (though the description clarifies it returns text, not JSON), the description is mostly complete. It covers purpose, usage, and output format, but could improve by mentioning behavioral aspects like idempotency or prerequisites.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the schema already documents both parameters (project_dir and profiles_dir) thoroughly. The description adds no additional parameter information beyond what's in the schema, maintaining the baseline score of 3 for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('install dbt package dependencies') and resource ('external packages'), distinguishing it from siblings like dbt_run or dbt_test that perform different operations. It explicitly mentions installing or updating dependencies, which is distinct from building, compiling, or testing the project.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool: 'when it needs to install or update external packages that the dbt project depends on.' It also explains the purpose ('ensures that all required modules, macros, and models from other packages are available before running the project'), which implicitly suggests using it as a prerequisite before other operations like dbt_run or dbt_test.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• dbt_lsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and does well by detailing output behavior based on parameters (verbose flag and output_format). It explains what gets returned in different scenarios, though it doesn't mention performance characteristics, error conditions, or authentication requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured with purpose first, then usage guidance, then detailed output behavior. Every sentence adds value, though the output format explanation is somewhat dense and could be more streamlined.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (8 parameters, no annotations, but with output schema), the description is complete. It covers purpose, usage context, and detailed output behavior. With an output schema present, it appropriately doesn't need to explain return value structure beyond the behavioral variations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all 8 parameters thoroughly. The description adds value by explaining how output_format and verbose parameters affect return values, but doesn't provide additional semantic context for other parameters beyond what's in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('dbt resources'), specifying it discovers models, tests, sources, and other resources. It distinguishes from siblings by focusing on discovery rather than execution operations like dbt_run or dbt_test.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use: 'when it needs to discover available models, tests, sources, and other resources within a dbt project' and explains the purpose: 'helps the agent understand the project structure, identify dependencies, and select specific resources for other operations'. This provides clear context for choosing this tool over execution-oriented siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• dbt_showA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and does well by clarifying this is a preview operation that doesn't modify the target database. It explains the non-destructive nature and default JSON output format, though it could mention performance characteristics or authentication requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured with a clear purpose statement followed by usage guidance and return format information. Every sentence adds value, and the information is front-loaded with the most important details about previewing without materialization.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity, no annotations, but with an output schema present, the description provides good context about the tool's purpose and behavior. It covers the key distinction from materialization tools and mentions the return format, though could benefit from more operational details given the lack of annotations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the baseline is 3. The description doesn't add parameter-specific information beyond what's already in the schema, though it does mention the default JSON format which relates to the 'output' parameter. No additional syntax or usage examples are provided for parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('preview the results of a model') and resource ('model'), distinguishing it from siblings by emphasizing non-materialization. It explicitly contrasts with tools like dbt_run that would materialize data, providing clear differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('when it needs to preview data from a specific model without materializing it') and offers concrete use cases ('inspect transformation results, debug issues, or demonstrate how data looks after processing'). It clearly distinguishes from materialization tools in the sibling set.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• dbt_testA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes the core behavior (running tests to validate data quality) and importantly discloses that the output is 'text (this command does not support JSON output format)', which is valuable behavioral information not captured elsewhere. It doesn't mention execution time, error handling, or side effects, but provides solid core behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured with three focused sentences: purpose statement, usage guidance, and output format disclosure. Every sentence adds value without redundancy. The information is front-loaded with the core purpose, followed by usage context, and ending with important behavioral detail about the output format.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a testing tool with 5 parameters, 100% schema coverage, and an output schema, the description provides good contextual completeness. It explains the purpose, when to use it, and importantly discloses the text-only output format. While it doesn't cover all possible behavioral aspects (like execution characteristics or error conditions), it provides sufficient context given the rich schema documentation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, with all 5 parameters well-documented in the schema. The description adds no specific parameter information beyond what's already in the schema, so it meets the baseline of 3. The description focuses on the tool's purpose and output format rather than parameter details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('run dbt tests'), the resource ('tests defined in a dbt project'), and the purpose ('validate data quality and integrity'). It distinguishes this tool from siblings by focusing specifically on testing rather than building, compiling, running, or other dbt operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool: 'when it needs to validate data quality and integrity by running tests defined in a dbt project.' It also explains the value: 'helps ensure that data transformations meet expected business rules and constraints before being used for analysis or reporting.' This gives clear context for when this testing tool should be selected over other dbt operations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• dbt_compileA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and does well: explains the tool is for preview/validation (not execution), discloses output format limitation ('does not support JSON output format'), and describes what the compilation process does. Could improve by mentioning performance implications or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Perfectly structured: purpose statement first, usage guidance second, return format third. Every sentence earns its place with zero waste. The two-sentence format is efficient and front-loaded with the most important information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Complete for a compilation tool: clear purpose, explicit usage guidelines, behavioral context (preview vs execution), and output format disclosure. With 100% schema coverage and an output schema present, the description focuses appropriately on the conceptual aspects rather than repeating structured data.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so parameters are well-documented in the schema. The description adds no specific parameter information beyond what's in the schema, but the schema documentation is comprehensive. Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Compile dbt models' with specific explanation of what compilation does ('generate the SQL that will be executed without actually running it'). It distinguishes from siblings like dbt_run (which executes) and dbt_show (which shows compiled SQL with sample data).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use: 'when it needs to generate the SQL that will be executed without actually running it' and provides three specific use cases (validating SQL syntax, previewing transformations, investigating dbt interpretations). This clearly differentiates from execution tools like dbt_run.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• dbt_debugA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes the tool's purpose (diagnostic validation), output format (text rather than JSON), and when it should be used (before running other operations). It doesn't mention potential side effects, performance characteristics, or error handling, but provides sufficient context for safe usage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured with two focused paragraphs: the first explains purpose and usage guidelines, the second clarifies output format. Every sentence adds value without redundancy, and the information is appropriately front-loaded with the most important context first.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's diagnostic nature, 100% schema coverage, and the presence of an output schema (implied by 'Has output schema: true'), the description provides complete context. It explains what the tool does, when to use it, and the output format, which is sufficient for this type of validation tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents both parameters. The description doesn't add any parameter-specific information beyond what's in the schema. This meets the baseline expectation when schema coverage is complete.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Run dbt debug to validate the project setup') and distinguishes it from siblings by focusing on troubleshooting configuration issues, checking connectivity, and verifying dependencies. It explicitly contrasts with tools like dbt_run or dbt_test that execute models rather than diagnose problems.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('when it needs to troubleshoot configuration issues, check database connectivity, or verify that all project dependencies are properly installed') and when not to use it ('before attempting to run models or tests'), clearly differentiating it from sibling tools that perform those execution tasks.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.