Server Quality Checklist

Tool Scores

• diagnose_order_delayA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the tool uses deterministic rules and returns supporting evidence, which is valuable. However, it does not explicitly state whether it is read-only, whether it fetches the context itself, or any side effects. For a diagnostic tool, this is acceptable but not fully transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that front-loads the action and provides all key details without redundancy. Every phrase earns its place, making it highly concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description lists the possible delay causes and mentions supporting evidence, but it does not clarify the exact input-output relationship (e.g., whether it fetches context internally) or describe the result format. Without an output schema, this ambiguity leaves room for misinterpretation, especially given the sibling get_order_context tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has only one parameter, orderId, with 0% schema description coverage. The description mentions 'an order's context' but does not explicitly define orderId. However, the parameter name is self-explanatory, and the phrase 'to an order's context' implies its use. Thus, the description adds minimal but sufficient meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: applying deterministic rules to an order's context to identify the most likely cause of a delay. It enumerates the possible outcomes (payment issue, inventory shortage, fulfillment delay/failure, cancelled, or no issue) and mentions supporting evidence, which distinguishes it from sibling tools like get_order_context that merely retrieve context.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when an order is delayed and the user needs to determine the cause. It does not explicitly state when not to use it or compare to alternatives, but the context is clear enough for an agent to select this tool over search_orders or recommend_resolution.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_ordersA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden and does disclose read-only behavior and the exclusion of general management. However, it does not describe return format, pagination, sorting, or any other behavioral details, leaving the agent with only a minimal safety profile.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences, front-loaded with the primary function and followed by a relevant safety note. Every word earns its place, and the structure is clean.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description should ideally specify return values. 'Find candidate orders' implies a list of matching orders, but the response structure is not stated. This minor gap is acceptable for a simple search tool, making it mostly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has no parameter descriptions (0% coverage), so the description must compensate. It states the parameters are optional filters by status and/or customer id, which adds meaning. It does not elaborate on enum semantics beyond the schema values, but for a simple search this is adequate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'Find candidate orders to investigate', using a specific verb and resource. It distinguishes itself from general order management and, combined with sibling names, implies a distinct investigative search function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description conveys clear usage context: it is for finding candidate orders to investigate, with optional filters. It explicitly states it is read-only and not for general management, but it does not name sibling alternatives or provide when-not-to-use guidance relative to them.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• recommend_resolutionA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the transparency burden. It describes an advisory, read-only-looking operation ('return recommended next actions') without explicitly stating side effects, permissions, or whether it modifies anything. It adds some context (for an operations specialist) but does not disclose any deeper behavioral traits beyond what the name implies.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core verb ('Diagnose') and immediately communicates both the diagnostic and recommendation actions. Every word contributes to understanding, with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity—one parameter, no output schema, and a clear purpose—the description is largely complete. It covers what it does and for whom, though it could briefly mention the format of the recommendations (e.g., a list of actions). This is a minor gap, not a major omission.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema only defines orderId as a string with zero descriptive coverage. The description ties orderId to 'an order's delay', providing meaningful context for the parameter. Although it doesn't specify format or how to obtain orderId, the single simple parameter is adequately explained by the description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: diagnose an order's delay and return recommended next actions. It uses a specific verb ('diagnose' and 'return') and resource ('order's delay', 'recommended next actions'), and it distinguishes itself from the sibling tool 'diagnose_order_delay' by adding the action recommendation component.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implicitly conveys when to use this tool—when you need recommended actions for an operations specialist. However, it does not explicitly contrast with alternatives like 'diagnose_order_delay', so it lacks explicit when-not guidance. The context is clear but lacks exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_order_contextA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosure. It clearly discloses the scope (facts only) and explicitly states what it does not do (no diagnosis), which is valuable behavioral context beyond a simple 'get' statement.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the action and content, and ending with a clear boundary. Every word earns its place; no fluff or repetition of schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple tool (one parameter, no output schema), the description provides enough detail on what is returned: the operational fact categories and the no-diagnosis boundary. It is complete for an AI agent to know when and how to invoke it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has only orderId with 0% coverage, but the description's 'for one order' makes the parameter's purpose self-evident. While it doesn't explicitly define orderId, the context of gathering information for a single order adequately compensates for the sparse schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Gather') and identifies a clear resource ('operational facts for one order') with enumerated facets. It distinguishes itself from siblings by stating it returns facts only, no diagnosis, positioning it as the evidence-gathering step.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context: it is for investigating why an order may be delayed. It also sets an exclusion ('no diagnosis') that indirectly points to diagnosing tools, but it does not explicitly name sibling alternatives such as diagnose_order_delay.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.