Server Quality Checklist

Tool Scores

• analyze_terraform_projectB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adds the fact that extraction is 'dynamic' but does not reveal important traits such as whether the tool is read-only, requires a local Terraform installation, accesses cloud APIs, or has limitations. This leaves significant gaps about side effects and operational behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loaded with the primary action, and contains no redundant filler. Every sentence contributes meaningful information about the tool's purpose and method.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has moderate complexity with one parameter and no output schema. The description captures the core purpose but omits details about expected output format, error scenarios, or operational requirements. It is minimally viable but not fully complete for a tool that processes user-supplied project paths.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for the single parameter ('Path to the project directory'). The description does not add details about parameter semantics beyond what the schema already provides, but it also does not conflict with or duplicate schema information. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb ('Analyze') and resource ('Terraform project') with a defined goal ('identify AWS services used'). It distinguishes itself from the sibling tool 'analyze_cdk_project' by explicitly targeting Terraform rather than CDK.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains what the tool does but does not explicitly state when to use it versus alternatives. There is no mention of exclusions, prerequisites, or comparison with 'analyze_cdk_project' or other siblings. Usage context is only implied by the tool's purpose.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_cost_reportA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosure. It does reveal important behaviors: generates markdown/CSV output, mandates inclusion of unit pricing, calculations, pricing model, and assumptions/exclusions. However, it does not explain how the report is returned (file write vs. string output), what happens with invalid pricing_data, or any side effects. This leaves gaps in understanding the tool's runtime behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with headings and bullet points, but it is quite lengthy, especially the example JSON. While informative, it could be more concise by trimming redundant statements like 'This tool requires AWS pricing data' since it is already implied by the first sentence and example. The example is valuable but adds bulk.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool complexity (10 params, nested objects) and lack of output schema, the description does not fully clarify the return format or behavior when output_file is not provided. It describes the report structure in general terms but leaves ambiguity about whether the tool returns the report as a string or saves to a file. More explicit mention of output delivery would improve completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Although schema description coverage is 100%, the description adds significant value through extensive example usage. It shows the exact structure for pricing_data, detailed_cost_data, and recommendations, which goes far beyond the schema's minimal descriptions. This is crucial for correctly constructing complex nested parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool generates a detailed cost analysis report for AWS services, which is a specific action on a specific resource. It distinguishes itself from sibling tools like get_pricing (raw data retrieval) and analyze_cdk_project/analyze_terraform_project (infrastructure analysis) by focusing on report generation from pricing data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description notes that the tool 'requires AWS pricing data', implying it should be used after get_pricing, but does not explicitly state when to use this tool versus alternatives or provide exclusion criteria. The example shows pricing_data from get_pricing, giving implicit context, but lacks a direct 'use this when...' statement.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_bedrock_patternsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden. 'Get' clearly implies a read-only operation, and the content scope is specified. However, it does not disclose potential caveats, authentication requirements, rate limits, or output structure. For a simple retrieval tool, this is acceptable but minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, well-structured sentence communicates purpose and scope in 13 words. It is front-loaded with the main action and includes necessary detail without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (no parameters, no output schema), the description adequately conveys what the tool returns: architecture patterns with component relationships and cost considerations. It is sufficient for an agent to know what to expect, though it could offer more detail on how the returned patterns are structured.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has effectively zero user-facing parameters (only the internal Context injection), so the baseline is 4. The description does not need to explain parameters, and the schema already documents the optional context. No additional parameter semantics are required.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Get') and resource ('architecture patterns for Amazon Bedrock applications'), and adds specific scope ('including component relationships and cost considerations'). It distinguishes from sibling tools like analyze_cdk_project and get_pricing by focusing on Bedrock architecture patterns.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention sibling tools or any selection criteria, leaving the agent to infer usage solely from the name and the generic 'Get' phrasing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_cdk_projectA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It adds a useful non-obvious detail: 'dynamically extracts service information from CDK constructs without relying on hardcoded service mappings.' This explains internal behavior. However, it does not disclose potential side effects (e.g., read-only vs. file modifications), output format, or prerequisites needed for analysis.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences: the first states the core purpose, and the second adds a relevant technical detail. It is front-loaded, concise, and contains no redundant or filler content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter tool, the description provides enough to select the tool and know what it does. However, it omits the return value/output format (e.g., list of service names) and any side effects or dependencies. Given the absence of annotations and an output schema, a bit more context would complete the picture.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with project_path described as 'Path to the project directory.' The tool description does not add any additional parameter semantics beyond what the schema already provides, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Analyze a CDK project to identify AWS services used.' It uses a specific verb ('analyze') with a specific resource ('CDK project') and an explicit outcome. It also distinguishes itself from the sibling analyze_terraform_project by specifying CDK, making the tool's scope unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for CDK projects, and the second sentence explains the method (dynamic extraction from CDK constructs), giving context. However, it does not explicitly mention when not to use this tool or name alternatives like analyze_terraform_project. The guidance is clear but relies on the sibling name to infer the Terraform alternative.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_pricingA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that only current pricing is available, the region parameter auto-creates a regionCode filter, large services require 2-3 filters to avoid 200+ record responses, and there is a default max_allowed_characters limit. However, it does not describe the response format or error behavior, leaving some gaps for a tool with no output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with headings, code blocks, bullet lists, and front-loaded purpose. However, it is verbose and repeats cost-optimization strategies multiple times, which could be condensed. The organization helps an agent navigate the content, but conciseness suffers from redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description could explicitly state the return format, but it thoroughly covers service discovery, filter construction, constraints, and use cases. The extensive workflow and examples provide sufficient context for correct invocation. Missing auth, error handling, and exact response structure are minor gaps for this API-oriented tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds substantial value: it explains service code discrepancies (e.g., 'AmazonES' vs 'AmazonOpenSearchService'), region auto-filter behavior, and the mandatory filter requirements for large services. Extensive code examples show exact filter structures and parameter usage, exceeding the schema baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Get detailed pricing information from AWS Price List API with optional filters', a clear verb+resource statement. It further defines the purpose as retrieving AWS pricing data for cost optimization, regional comparisons, compliance reporting, and research, distinguishing it from sibling analysis and reporting tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit when-to-use guidance: use get_price_list_file for historical data, avoid savings plans/spot pricing, and apply specific filters for large services. It includes anti-patterns, success criteria, and detailed use-case examples, giving the agent clear direction on when and how to invoke the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.