Server Quality Checklist

Tool Scores

• run-aws-codeC
  2.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure but offers minimal information. 'Run AWS code' suggests execution but doesn't disclose whether this is a read or write operation, what permissions are required, whether it has side effects, rate limits, or error handling behavior. The description fails to provide the behavioral context needed for safe and effective tool invocation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise at just two words, with zero wasted language. While this conciseness comes at the expense of completeness, from a pure structural perspective, it's front-loaded and contains no unnecessary verbiage. Every word in the description directly relates to the tool's function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of running AWS code (which involves execution, potential mutations, security implications, and error handling), the description is completely inadequate. With no annotations, no output schema, and a minimal description, an agent lacks critical information about what this tool actually does, what it returns, and how to use it safely. The description fails to provide the contextual completeness needed for a tool of this nature.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the input schema already documents all four parameters thoroughly. The description adds no additional parameter semantics beyond what's already in the schema. The baseline score of 3 reflects that the schema does the heavy lifting, though the description could have provided higher-level context about how parameters relate to each other or typical usage patterns.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Run AWS code' is essentially a tautology that restates the tool name without providing meaningful specificity. It doesn't clarify what type of AWS code is run, what resources it interacts with, or what distinguishes it from sibling tools like list-credentials and select-profile. The description lacks a clear verb+resource combination that would help an agent understand the tool's actual function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines1/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides absolutely no guidance about when to use this tool versus alternatives. There's no mention of appropriate contexts, prerequisites, or relationships to sibling tools. An agent would have no information about whether this tool should be used for AWS operations versus the other available tools, making selection decisions difficult.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list-credentialsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions that credentials are 'configured/usable on this machine', hinting at local scope, but fails to detail critical aspects like whether this is a read-only operation, potential side effects, error handling, or output format, which are essential for safe and effective use.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's function without unnecessary words. It is front-loaded with the core action and resource, making it easy to parse and understand quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema) and lack of annotations, the description is minimal. While it covers the basic purpose, it omits important contextual details such as the output structure, potential errors, or how it interacts with sibling tools, making it incomplete for reliable agent operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and the schema description coverage is 100%, so there is no need for parameter documentation in the description. The description appropriately focuses on the tool's purpose without redundant parameter details, aligning with the baseline expectation for parameterless tools.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List all') and the resource ('AWS credentials/configs/profiles'), making the tool's purpose evident. However, it does not explicitly differentiate from sibling tools like 'select-profile', which might involve similar resources but different actions, leaving room for minor ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives such as 'select-profile' or 'run-aws-code'. It lacks context about prerequisites, scenarios for usage, or exclusions, leaving the agent without clear direction on tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• select-profileA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses the tool's stateful nature (affects subsequent interactions) and conditional SSO authentication, which are valuable behavioral traits. However, it doesn't mention potential side effects, error conditions, or what happens if authentication fails.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly concise with two sentences that each earn their place. The first sentence states the core function, and the second adds important conditional behavior. There's zero wasted language or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with 2 parameters, 100% schema coverage, and no output schema, the description provides adequate context about the tool's purpose and behavioral characteristics. The main gap is the lack of information about return values or what constitutes successful execution.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, providing complete parameter documentation. The description adds no additional parameter semantics beyond what's in the schema, so it meets the baseline score of 3 where the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('selects', 'does SSO authentication') and identifies the resource ('AWS profile'). It distinguishes from sibling tools by focusing on profile selection rather than credential listing or code execution.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context about when to use this tool ('for subsequent interactions') and mentions SSO authentication as a conditional behavior. However, it doesn't explicitly state when NOT to use it or name specific alternatives among the sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.