Server Quality Checklist

Tool Scores

• create-userC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool creates a user, implying a write/mutation operation, but doesn't disclose any behavioral traits such as permissions required, whether the operation is idempotent, error handling, or what happens on success/failure. This leaves significant gaps for an agent to understand how to use it safely and effectively.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence that is front-loaded with the core action. There is no wasted verbiage or unnecessary elaboration, making it highly efficient and easy to parse. Every word earns its place by conveying essential information without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a user creation tool with 5 required parameters, no annotations, and no output schema, the description is incomplete. It lacks details on behavioral aspects, parameter usage, and expected outcomes. While concise, it doesn't provide enough context for an agent to confidently invoke the tool without additional assumptions or errors.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate by explaining parameters, but it adds no meaning beyond the schema. The description mentions 'realm' but doesn't clarify what a realm is or how parameters like 'username' or 'email' are used. With 5 required parameters and no guidance in the description, this is inadequate for understanding parameter roles.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Create') and resource ('new user in a specific realm'), making the purpose immediately understandable. It distinguishes from sibling tools like 'delete-user' and 'list-users' by specifying creation rather than deletion or listing. However, it doesn't specify what kind of user or system this applies to, which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., needing realm access), when not to use it (e.g., for existing users), or direct alternatives among siblings. The phrase 'in a specific realm' hints at context but lacks explicit usage instructions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• delete-userC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It indicates a destructive operation ('Delete'), but fails to specify whether deletion is permanent, reversible, requires admin permissions, or has side effects. This leaves significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, direct sentence with no wasted words, making it highly concise and front-loaded. It efficiently communicates the core action without unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a destructive tool with no annotations, 0% schema coverage, and no output schema, the description is insufficient. It lacks details on permissions, irreversibility, error handling, and return values, leaving critical context gaps for safe and effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate by explaining parameters. It mentions 'realm' and 'userId' implicitly but doesn't define their meaning, format, or constraints. This adds minimal value beyond the bare schema, failing to adequately clarify parameter semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Delete') and resource ('a user from a specific realm'), making the purpose unambiguous. However, it doesn't differentiate from sibling tools like 'create-user' or 'list-users', which would require explicit comparison to achieve a score of 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'create-user' or 'list-users', nor does it mention prerequisites, permissions, or consequences. It simply states what the tool does without contextual usage information.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list-usersC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It states it's a list operation (implying read-only), but doesn't mention permissions needed, pagination behavior, rate limits, or what the output looks like. For a tool with zero annotation coverage, this leaves significant behavioral gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero wasted words. It's appropriately sized for a simple list tool and front-loads the essential information ('List users'). Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, 0% schema coverage, and no output schema, the description is insufficiently complete. It covers the basic purpose but lacks parameter details, behavioral context, and output information that would help an agent use this tool effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the single parameter 'realm' is undocumented in the schema. The description adds minimal value by mentioning 'specific realm', but doesn't explain what a realm is, provide format examples, or clarify if it's a name, ID, or something else. It doesn't compensate for the schema coverage gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List') and resource ('users in a specific realm'), making the purpose immediately understandable. It doesn't explicitly distinguish from sibling tools like 'list-realms' or 'create-user', but the specificity of 'users' vs 'realms' provides implicit differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'list-realms' or 'create-user'. It mentions 'specific realm' which hints at a prerequisite, but offers no explicit usage context, exclusions, or comparison to sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list-realmsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It states what the tool does ('list all available realms') but doesn't describe return format, pagination, permissions needed, or any side effects. For a tool with zero annotation coverage, this is insufficient to inform the agent about how the tool behaves beyond its basic function.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that states the tool's purpose without any wasted words. It's appropriately sized for a simple list operation and front-loaded with the essential information. Every part of the description earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete. It doesn't explain what 'realms' are, what the return format looks like, or any behavioral traits. For a tool with no structured data to rely on, the description should provide more context to help the agent understand the tool fully.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and schema description coverage is 100% (though empty). The description doesn't need to compensate for missing parameter info. A baseline of 4 is appropriate since there are no parameters to document, and the description accurately reflects this by not mentioning any.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('list') and resource ('realms'), making it immediately understandable. However, it doesn't differentiate from sibling tools like 'list-users', which could cause confusion about scope. It's not tautological or misleading, but lacks sibling distinction for a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'list-users' or other siblings. There's no mention of prerequisites, context, or exclusions. While the purpose is clear, the absence of usage context leaves the agent without direction for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.