Server Quality Checklist

Tool Scores

• run-aws-codeC
  2.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure but offers minimal information. 'Run AWS code' doesn't reveal whether this is a read or write operation, what permissions are required, whether it has side effects, or what happens on failure. The extensive rules in the schema's code parameter description are implementation requirements rather than behavioral characteristics of the tool itself.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise at just three words, with zero wasted space. It's front-loaded with the essential information (though that information is inadequate). There are no unnecessary sentences or redundant phrasing to critique from a conciseness perspective.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 4 parameters, no annotations, and no output schema, the description is completely inadequate. It doesn't explain what the tool does, when to use it, what behaviors to expect, or what results it produces. The extensive rules in the schema's code parameter description don't compensate for the missing contextual information about the tool's purpose and operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds no parameter semantics beyond what's already in the schema, which has 100% coverage with detailed descriptions for all four parameters. The baseline score of 3 reflects that the schema does the heavy lifting, though the tool description itself contributes nothing additional about parameter meaning or usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Run AWS code' is a tautology that merely restates the tool name without specifying what the tool actually does. It doesn't explain that this tool executes JavaScript code against AWS environments using the AWS SDK, nor does it differentiate from sibling tools like list-credentials or select-profile. The description fails to provide a clear verb+resource combination that would help an agent understand the tool's function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines1/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. There's no mention of prerequisites, appropriate contexts, or exclusions. While the input schema's code parameter description contains extensive implementation rules, these are technical constraints rather than usage guidelines for an AI agent deciding when to invoke this tool versus other AWS-related tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list-credentialsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It implies a read-only operation by using 'List,' but doesn't specify if it requires permissions, how it handles errors, or what the output format looks like. For a tool with zero annotation coverage, this lack of detail on behavior is a notable shortfall.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's function without any fluff. It is front-loaded with the core action and resource, making it easy to parse. Every word contributes to understanding the purpose, earning a top score for conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema, no annotations), the description is adequate but has clear gaps. It explains what the tool does but lacks details on usage guidelines, behavioral traits, and output format. For a basic list tool, this is minimally viable but could be more complete by addressing when to use it and what to expect in return.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and schema description coverage is 100%, so there are no parameters to document. The description doesn't need to add parameter semantics, but it does imply the scope ('on this machine'), which is useful context. Baseline for 0 parameters is 4, as the description adequately covers the tool's intent without unnecessary parameter details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'List all AWS credentials/configs/profiles that are configured/usable on this machine.' It specifies the verb ('List') and resource ('AWS credentials/configs/profiles'), making the action explicit. However, it does not explicitly differentiate from sibling tools like 'select-profile' or 'run-aws-code', which could involve similar resources, so it doesn't reach a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools or contexts where this tool is preferred, such as for inventory purposes before selecting a profile. Without any usage context, the agent must infer when to invoke it, which is a significant gap.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• select-profileA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adds useful context about SSO authentication ('If needed, does SSO authentication'), which is not inferable from the input schema. However, it lacks details on side effects (e.g., whether this persists across sessions), error handling, or what 'subsequent interactions' specifically entails, leaving gaps in behavioral understanding.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise (two short sentences) and front-loaded with the primary purpose. Every word earns its place, with no redundant or vague phrasing, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (configures context for AWS operations) and lack of annotations or output schema, the description is mostly complete. It covers the purpose, conditional SSO behavior, and usage context. However, it misses details on return values or error cases, which could be important for a tool that sets state, slightly reducing completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, so the schema already fully documents both parameters ('profile' and 'region'). The description does not add any meaning beyond what the schema provides (e.g., it doesn't explain profile naming conventions or region implications), resulting in the baseline score of 3 for adequate but no extra value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Selects AWS profile') and resource ('for subsequent interactions'), with explicit mention of SSO authentication as a conditional behavior. It distinguishes from sibling tools like 'list-credentials' (which lists rather than selects) and 'run-aws-code' (which executes code rather than configuring context).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool ('for subsequent interactions') and implies it should be used before other AWS operations. However, it does not explicitly state when not to use it or name alternatives (e.g., when to use 'list-credentials' instead), which prevents a perfect score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.