Server Quality Checklist

Tool Scores

• RunCheckovScanA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes the tool as running a scan and returning results, which implies a read-only operation, but does not detail execution characteristics such as runtime, error handling, or external dependencies (e.g., Checkov installation). It adds context by linking to Checkov's website and noting it's open-source, but lacks specifics on permissions or side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized, with a clear purpose statement followed by parameter and return sections. It avoids redundancy, though the link to Checkov's website, while informative, could be considered slightly extraneous. Overall, it is efficient and front-loaded with key information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (security scanning with multiple parameters), the description is reasonably complete. It explains the tool's purpose, parameters, and return value, and an output schema exists to detail results. However, it could improve by addressing usage guidelines and behavioral aspects like execution limits or error scenarios, which are not covered by annotations or schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description lists parameters with brief explanations (e.g., 'Directory containing Terraform files to scan'), but does not add significant meaning beyond what the schema provides, such as examples or constraints. The baseline score of 3 reflects adequate coverage without extra value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verb ('run') and resource ('Checkov security scan on Terraform code'), distinguishing it from sibling tools like ExecuteTerraformCommand or ExecuteTerragruntCommand. It explicitly mentions scanning for 'security and compliance issues' and 'identifying potential vulnerabilities and misconfigurations,' providing a precise scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description lacks explicit guidance on when to use this tool versus alternatives. While it mentions scanning Terraform code, it does not specify prerequisites (e.g., whether Terraform files must be valid), exclusions (e.g., non-Terraform files), or comparisons to sibling tools like ExecuteTerraformCommand, which might handle other Terraform operations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ExecuteTerragruntCommandA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and does well by disclosing key behavioral traits: it executes commands that can be destructive (apply, destroy), mentions AWS account targeting, describes Terragrunt's extended functionality, and notes parameter constraints (terragrunt_config not valid with run-all). However, it doesn't cover authentication requirements, rate limits, or error handling specifics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with purpose statement, context about Terragrunt, parameter list, and return information. It's appropriately sized for a 9-parameter tool but could be more front-loaded; the Terragrunt explanation paragraph could be shortened. Most sentences earn their place, though some parameter explanations are redundant with schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (9 parameters, potential destructive operations) and absence of annotations, the description does reasonably well. It explains the tool's purpose, lists parameters, mentions return type, and provides Terragrunt context. However, with no annotations and significant behavioral implications (destructive commands), it should ideally cover more about authentication, safety warnings, or execution environment.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description lists all parameters with brief explanations but adds minimal semantic value beyond what's already in the schema descriptions. The note about 'terragrunt_config not valid with run-all' is the only significant addition, but most parameter explanations simply restate schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Execute Terragrunt workflow commands against an AWS account' with specific verbs (init, plan, validate, apply, destroy, run-all) and distinguishes it from sibling tools by mentioning Terragrunt's unique features like remote state management and multi-module execution. It differentiates from ExecuteTerraformCommand by focusing on Terragrunt workflows.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context through the mention of Terragrunt extending Terraform and AWS account targeting, but lacks explicit guidance on when to use this tool versus alternatives like ExecuteTerraformCommand or the various search tools. No when-not-to-use guidance or clear prerequisites are provided beyond the AWS context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• SearchUserProvidedModuleA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes what the tool does (analyzes modules) and mentions the return type, but does not cover aspects like rate limits, authentication needs, or error handling. It adds some context but lacks comprehensive behavioral traits beyond the basic operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the purpose, followed by examples and parameter details. It is appropriately sized, but could be slightly more concise by reducing redundancy in parameter explanations. Every sentence adds value, though some information overlaps with the schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity, 100% schema coverage, and the presence of an output schema, the description is mostly complete. It explains the purpose, usage, and parameters adequately, but could improve by addressing behavioral aspects like permissions or limitations. The output schema reduces the need to detail return values, making it sufficient but not exhaustive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters. The description adds minimal value by listing parameters and providing examples, but does not explain semantics beyond what the schema provides, such as format details or constraints. The baseline score of 3 is appropriate as the schema handles most documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Search for a user-provided Terraform registry module and understand its inputs, outputs, and usage.' It specifies the verb ('search'), resource ('Terraform registry module'), and scope ('analyzes its input variables, output variables, README, and other details'), distinguishing it from sibling tools like ExecuteTerraformCommand or RunCheckovScan, which perform different actions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool, such as analyzing module details, but does not explicitly state when not to use it or name specific alternatives. It implies usage for understanding module structures, which helps differentiate from siblings like SearchAwsProviderDocs, but lacks explicit exclusions or comparisons.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ExecuteTerraformCommandA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that the tool executes commands (implying potential mutations like apply/destroy) and mentions output handling (strip_ansi), but does not cover critical behavioral aspects such as authentication requirements, rate limits, error handling, or safety warnings for destructive commands like destroy. It adds some context but leaves significant gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (overview, parameters, returns), front-loaded with the core purpose, and every sentence adds value without redundancy. It efficiently covers essential information in a compact format.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (5 parameters, mutation capabilities, no annotations) and the presence of an output schema (returns TerraformExecutionResult), the description is mostly complete. It covers purpose, parameters, and output type, but lacks details on behavioral traits (e.g., auth, safety) that would be crucial for a tool with commands like destroy, preventing a perfect score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds value by listing all parameters with brief explanations in a structured 'Parameters' section, clarifying their roles beyond the schema. However, it does not provide deep semantic insights (e.g., format examples for variables or region implications), keeping it at a 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool executes Terraform workflow commands against an AWS account, specifying the exact commands (init, plan, validate, apply, destroy) and distinguishing it from sibling tools like ExecuteTerragruntCommand and RunCheckovScan. It provides a specific verb ('execute') and resource ('Terraform workflow commands') with clear scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for Terraform operations in AWS contexts but does not explicitly state when to use this tool versus alternatives like ExecuteTerragruntCommand or RunCheckovScan. It provides basic context (working with Terraform in AWS) but lacks explicit guidance on exclusions or comparative scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• SearchAwsProviderDocsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does well by disclosing behavioral traits such as handling prefixes automatically (e.g., 'aws_s3_bucket' or 's3_bucket'), specifying valid values for asset_type, and describing the return format. It does not mention rate limits or authentication needs, but covers key operational aspects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, starting with a clear purpose, followed by usage details, examples, and parameter explanations. Every sentence adds value without redundancy, making it efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity and the presence of an output schema (which details return values), the description is complete enough. It covers purpose, usage, parameters, and behavioral aspects, providing sufficient context for an AI agent to understand and invoke the tool correctly without needing to repeat output details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters thoroughly. The description adds some value by explaining asset_type options and providing examples, but does not significantly enhance semantics beyond what the schema provides, aligning with the baseline for high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches AWS provider documentation for resources and attributes, specifying it retrieves comprehensive details like descriptions, code snippets, and references. It distinguishes from siblings like ExecuteTerraformCommand by focusing on documentation search rather than execution or scanning.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context on when to use this tool (e.g., for searching documentation) and includes examples for different scenarios. However, it does not explicitly state when not to use it or compare it to alternatives like SearchAwsccProviderDocs, which might handle different documentation sources.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• SearchAwsccProviderDocsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and does well by detailing what the tool returns (list of documentation entries with specific details like URLs and schema information). It doesn't mention rate limits, authentication needs, or error handling, but covers the core behavior adequately for a search tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with purpose but includes repetitive examples and a verbose 'Returns' section that overlaps with output schema. Sentences like 'The tool will automatically handle prefixes' are useful, but overall it could be more streamlined by removing redundancy and focusing on unique guidance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (searching documentation with two parameters), rich output schema (implied by 'Returns' section), and no annotations, the description is complete. It covers purpose, usage, parameters, and return details, making it sufficient for an AI agent to understand and invoke the tool correctly without gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds value by explaining parameter usage with examples (e.g., handling prefixes, valid values for asset_type) and clarifying that some resources and data sources share names, which enhances understanding beyond the schema's basic descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches AWSCC provider documentation for resources and attributes, specifying it retrieves comprehensive details like descriptions, code snippets, and schema references. It distinguishes from sibling tools like SearchAwsProviderDocs by focusing on AWSCC (Cloud Control API) rather than standard AWS provider documentation, avoiding tautology.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly provides when to use this tool with examples for different asset types (resource, data_source, both) and clarifies it handles prefixes automatically. It distinguishes from alternatives by specifying the AWSCC provider context, though it doesn't explicitly name when not to use it versus siblings like SearchAwsProviderDocs, but the context makes this clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• SearchSpecificAwsIaModulesA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure and does so effectively. It explains what information is returned (README content, variables.tf content, submodules), what the search covers (module names, descriptions, README content, variable definitions), and provides concrete examples of how the query parameter affects results. The only minor gap is lack of explicit mention of error conditions or rate limits.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections: purpose statement, module listing, return details, examples, and parameters. While somewhat lengthy, every sentence adds value by explaining scope, behavior, or usage. The examples are particularly helpful and efficiently demonstrate different use cases.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (true), the description doesn't need to explain return values in detail, yet it still provides a comprehensive list of what information is returned. With no annotations, the description fully covers the tool's behavior, scope, and usage. The examples bridge any potential gaps, making this complete for a search tool with one parameter.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the baseline is 3. The description adds significant value by explaining that the query parameter is 'Optional' (contradicting the schema's 'required' designation), providing multiple concrete examples of query usage, and clarifying that an empty query returns all four modules. This goes well beyond what the schema provides, though it doesn't fully resolve the schema contradiction.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches for 'specific AWS-IA Terraform modules' and lists the exact four modules it targets (aws-ia/bedrock/aws, aws-ia/opensearch-serverless/aws, aws-ia/sagemaker-endpoint/aws, aws-ia/serverless-streamlit-app/aws). It distinguishes itself from sibling tools like SearchAwsProviderDocs and SearchUserProvidedModule by specifying this fixed set of AWS-IA modules rather than general searches.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool through multiple examples: 'To get information about all four modules', 'To find modules related to Bedrock', 'To find modules related to vector search', and 'To find modules with specific configuration options'. It also clarifies that an empty query returns all four modules, establishing clear usage patterns.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.