Server Quality Checklist

Tool Scores

• get_billing_statusA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It indicates the operation is a read (取得する) and details the response contents, including the list of statuses and the inclusion of payment_instructions for unpaid orders. It does not mention error conditions or permissions, but covers the core behavior well.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the action, and contains no filler. Every clause adds value, specifying the input (user ID), what is retrieved (billing statuses), and the special case of unpaid orders (payment instructions).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter read tool with no output schema, the description adequately explains the return value (list of statuses) and the conditional inclusion of payment instructions. It does not cover error cases or edge situations (e.g., missing user), but for this level of complexity, it is reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already provides 100% coverage for the single parameter, describing user_id as the registered email address. The description repeats the email reference but adds no new semantic information beyond what the schema offers, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves a list of billing statuses (unpaid/paid) for all orders of a given user, using a specific verb (取得する) and resource (billing status list). It distinguishes from siblings like lookup_order or process_return by focusing on billing status, not order details or returns.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for billing status inquiries and provides clear context, but it does not explicitly state when to use this tool versus alternatives or mention any exclusions. No reference to sibling tools is made, so the guidance is implicit rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lookup_orderA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that the tool is used for identity confirmation and lists the output fields, but does not describe error behavior, permissions, or any side effects. For a lookup tool, this is moderate but not rich disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences, front-loaded with the action, input, and output. Every word earns its place, with no unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter lookup tool with no annotations and no output schema, the description covers input, output fields, and usage context. It lacks error/edge-case details, but the field list is provided, making it reasonably complete for the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The sole parameter user_id is fully described in the schema as an email address, and the description repeats this without adding new meaning. Since schema coverage is 100%, the description does not compensate or enhance parameter understanding beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves a user's order list (product name, quantity, order date, billing status) based on user ID (email). It uses a specific verb (取得する) and resource (注文一覧), and is distinct from sibling tools like check_return_eligibility or get_billing_status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly frames the tool as the first step in handling an inquiry, also serving to confirm user identity. This provides clear context for when to use it, though it does not explicitly mention when not to use alternatives or name them as exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_return_eligibilityA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description takes on the full burden of behavioral disclosure. It notably explains that expired or already-returned orders yield a normal response with eligible=false rather than an error, which is crucial non-obvious behavior. It does not cover all edge cases (e.g., invalid order ID), but the disclosed behavior is valuable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exactly two sentences: the first states the primary function, and the second clarifies the non-error response behavior. Every word earns its place, with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (1 parameter, no output schema), the description sufficiently covers the main logic and the key edge case. It could be more explicit about the response format (e.g., type of eligible field), but it provides enough for an agent to understand and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already provides full documentation for the single parameter order_id, including type, minLength, and an example. The description merely references '注文ID' without adding new semantic details, so credit is limited to the schema's full coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: given an order ID, it determines return eligibility based on whether the order date is within one month. This distinct verb-resource pairing (判定する + 返品可否) sets it apart from siblings like process_return or lookup_order.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool (when you need to check return eligibility) and clearly differentiates it from alternatives like process_return by focusing on the determination step. However, it does not explicitly state when not to use it or mention alternative tools by name.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• process_returnA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It discloses that the tool sends a return acceptance email (simulated) and returns an error for ineligible orders. It doesn't detail idempotency, return response structure, or other side effects, but the key behaviors are well covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, consisting of two sentences. It front-loads the core action in the first sentence and adds important side effects and error handling in the second. No superfluous information is present.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the absence of an output schema, the description adequately covers the prerequisite, the main action, the email side effect, and the error condition. It doesn't specify the exact success response format, but for a sample tool it is sufficiently complete for an agent to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for both required parameters, so the baseline is 3. The description adds minimal meaning beyond the schema (e.g., 'receives return reason' is essentially the same as the schema's description). No additional parameter details are provided.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: confirming return processing for orders already judged returnable by check_return_eligibility. It also distinguishes itself from siblings by referencing this prerequisite and by specifying the action (confirming the return) and the side effect (sending notification email).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage context: it should be used for orders that have passed check_return_eligibility. It also warns that calling it on non-eligible orders results in RETURN_NOT_ELIGIBLE error, implying when not to use it. However, it doesn't explicitly mention alternatives or when to choose other tools like escalate_to_human.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• escalate_to_humanA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full behavioral burden. It discloses server-side turn management, status transitions at 5 turns (end confirmation) and 10 turns (forced termination with request to create a separate inquiry), and that it does not generate reply text. However, it omits details about session creation, error handling, or full response structure, so some ambiguity remains.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences with no fluff: first sentence defines scope, second gives invocation frequency, third explains turn limits and the lack of response generation. It is well-structured and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no output schema, so the description compensates by mentioning the status field and turn limits, but it does not enumerate all possible status values or other response fields. It also doesn't explain how to start a session if session_id is new. Still, for a simple 2-param tool, it provides enough to use it correctly in most cases.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Both parameters are fully described in the schema (100% coverage), so the baseline is 3. The description does not add parameter-specific semantics beyond what the schema already provides; it only reiterates that the tool is called per utterance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it manages conversation sessions with a human operator for inquiries that do not fall under returns or billing ('返品にも請求にも該当しない問い合わせについて'). This explicitly scopes the tool's purpose and distinguishes it from sibling tools like lookup_order, process_return, and get_billing_status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives explicit usage guidance: 'ユーザーの発言1回ごとに呼び出すこと' (call it for each user utterance) and excludes returns/billing cases, implying when to use alternatives. It also instructs that the tool does not generate operator replies, so the calling agent must craft responses, providing clear context for correct invocation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.