Server Quality Checklist

Tool Scores

• create_support_ticketB
  3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden for behavioral disclosure. It only notes the auth requirement ('requires ticket:write') but does not mention side effects, rate limits, or other behavioral traits beyond the obvious creation action.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, consisting of a single sentence. While it is not verbose, it is barely adequate and could benefit from additional context without becoming unwieldy. It is front-loaded with the core action but lacks structure.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (4 parameters, no nested objects, presence of output schema) and the absence of annotations, the description is incomplete. It fails to explain the result of creating a ticket, how parameters interact, or any required order of operations, leaving an agent with insufficient context to use the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, meaning the description offers no explanation of the parameters. The input schema provides names and types but lacks descriptions, and the tool description adds no additional semantic meaning to help an agent correctly populate the subject, priority, customer_id, or description fields.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'Create a customer support ticket,' which is a specific verb+resource. It clearly distinguishes the tool from siblings like process_order and update_inventory, which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions the required permission 'ticket:write,' providing context for when to use this tool. However, it lacks explicit guidance on when not to use it or alternatives, missing an opportunity to clarify when other tools might be more appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• process_orderB
  3.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses the atomic write operation, inventory decrement side effect, and required permission (order:write). It does not detail failure behavior or idempotency, but for a mutation tool, this is substantive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence plus a permission note, with no extraneous words. It is front-loaded with the main purpose and concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is non-trivial with four required parameters, atomic inventory decrement, and an output schema (not shown). The description omits details on idempotency key usage, return value, and exact inventory behavior. More context is needed for a complete understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0% and the description does not mention any of the four required parameters (customer_id, sku, quantity, idempotency_key). The description fails to add any meaning beyond the schema, which is insufficient for a tool with no parameter descriptions in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool creates an order atomically and decrements inventory, with a specific verb and resource. It distinguishes itself from siblings like create_support_ticket (different resource) and update_inventory (different action).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions a required permission (order:write) but provides no guidance on when to use this tool versus alternatives, such as when to use update_inventory separately or create_support_ticket. There is no context about exclusions or prerequisites beyond the permission.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• update_inventoryA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description adds 'validation and audit logging' as behavioral traits. However, it does not specify what validation occurs, the nature of audit logging, or whether the operation is destructive or reversible. More detail is needed for a tool without annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences, front-loaded with the core action. Every phrase adds value: the action, key features (validation, audit logging), and permission requirement. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Although an output schema exists (not shown), the description does not explain the return value or what the agent can expect. It lacks detail on how 'adjust' works (e.g., whether quantity_delta is additive or absolute), the role of 'reason', and potential side effects. Given the critical nature of inventory operations, this is insufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It fails to explain any of the three parameters (sku, quantity_delta, reason). It does not clarify that quantity_delta is a positive/negative integer or that reason is a required free-text field, leaving the agent to infer from names alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'Adjust inventory' with specific verb and resource, and adds validation, audit logging, and permission requirements. It clearly distinguishes from sibling tools like create_support_ticket and process_order.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions the required permission 'inventory:write', providing a precondition. It implies usage for inventory adjustments, but lacks explicit when-to-use or when-not-to-use guidance or alternatives beyond the permission context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.