Server Quality Checklist

Tool Scores

• chatbuy_statusC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full responsibility for disclosing behavior. It only says 'check login status' and doesn't reveal whether the operation is read-only, what the response looks like, or any prerequisites or side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short lines, front-loaded with the core purpose, and repeated in Japanese and English. Every word earns its place, and there is no unnecessary fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Although the tool is simple, there is no output schema and no annotations, so the description should at least indicate what the status result looks like. It doesn't mention return values, success/failure behavior, or how login status is determined, leaving significant gaps for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema fully documents the single optional 'site' parameter, so schema coverage is 100%. The description adds no additional meaning about the parameter, but the schema already handles it adequately.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('check') and resource ('login status') for a shopping site, clearly stating what the tool does. It is distinguishable from siblings like 'chatbuy_login' because 'status' implies inspection rather than performing a login, though it doesn't explicitly call out the difference.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given about when to use this tool versus alternatives such as chatbuy_login or chatbuy_activate. The only context is 'for a shopping site,' which doesn't help an agent choose this tool over its siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• chatbuy_searchB
  3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It states the core function (converts a natural-language request into candidate items with prices) but doesn't disclose whether this is a read-only operation, whether it requires authentication, if it has side effects (e.g., making external purchases), or any rate limits. It also doesn't describe the nature of the returned candidates (e.g., count, ordering, source). This lack of behavioral context is a notable gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, consisting of two short sentences (one in Japanese, one in English) that immediately convey the purpose. There is no fluff, filler, or redundant information. The bilingual format is acceptable and doesn't add unnecessary length. It is appropriately sized for a tool of this complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 2 parameters, no annotations, and no output schema, the description should provide more context. It does mention the return format ('candidate items with prices'), but it leaves out essential details like what 'site' controls, whether the tool requires prior login/activation, and the exact shape or limitations of the search results. For a simple tool, the description is partially informative but not complete enough for an agent to invoke it correctly without guessing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is only 50%: the 'prompt' parameter has a description with an example, but the 'site' parameter has no description at all. The tool description doesn't compensate for this gap—it never mentions the 'site' parameter or its meaning/possible values. Thus, one of the two parameters is entirely undocumented, and the description fails to add meaningful insight beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Turn a natural-language shopping request into candidate items with prices') and a clear resource (shopping request). It clearly communicates what the tool does, but it doesn't explicitly distinguish itself from sibling tools like chatbuy_login or chatbuy_review, though the function is obviously distinct. Since it doesn't explicitly compare with siblings, I deduct one point from full marks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this tool is for converting shopping requests into product candidates, but it doesn't explicitly state when to use it vs alternatives. There's no mention of prerequisites, such as requiring login or activation, or exclusions (e.g., when not to use it). The usage context is implied by the tool's name and purpose, but no explicit guidance is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• chatbuy_loginA
  3.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description discloses key behaviors: it opens a real browser, requires human interaction for login, and stores the session locally. It also notes the session persists, which is important for understanding state. However, it does not explain what happens if the session expires.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences, in both Japanese and English, making the purpose clear without unnecessary detail. Each sentence contributes to understanding the tool's behavior.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  While the tool's basic behavior is explained, the single parameter 'site' is completely unexplained, and there is no information about return values or error conditions. For a tool with minimal schema detail, this is a significant gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema includes a 'site' parameter with no description and the tool description does not explain what 'site' refers to. With 0% schema coverage, the description fails to compensate, leaving the agent without guidance on what value to pass.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool opens a real browser to the login page, with a specific verb and resource. It distinguishes itself from siblings like chatbuy_search and chatbuy_status by focusing on the manual login step.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this is a one-time manual login setup that persists locally, but it does not explicitly state when to use it relative to other tools or any exclusions. The guidance is implied rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• chatbuy_activateA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It only mentions the effect (removing the limit) but does not disclose potential side effects (e.g., billing implications), prerequisites (e.g., active Stripe subscription), error behavior for unregistered emails, or reversibility. For a mutation tool, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences, one in Japanese and one in English, both immediately conveying the action and effect. There is no filler or redundant content, and the bilingual format is compact while serving localization needs.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a low-complexity tool with one parameter and no output schema, the description adequately covers purpose, effect, and parameter semantics. It lacks preconditions and error handling details, but these are less critical given the simplicity. The description is complete enough for an agent to select and invoke the tool in most cases.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with a clear parameter description ('Stripeサブスク登録に使ったメールアドレス' – email used for Stripe subscription registration). The tool description adds 'registered email' context, but the schema already provides the essential meaning. This matches the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool activates the optional supporter subscription by email and removes the free-tier search limit. This distinguishes it from sibling tools like chatbuy_status, chatbuy_login, chatbuy_search, and chatbuy_review. The verb 'activate' and resource 'supporter subscription' are explicit and specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when a registered email exists and the user wants to remove the free-tier limit, but it does not explicitly state when to use this tool versus alternatives or when not to use it (e.g., if already activated). No alternative sibling tools are mentioned, nor any prerequisites like prior subscription setup.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• chatbuy_reviewA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full behavioral disclosure burden. It transparently states the key boundary—never clicking the final purchase button—and describes the actions it takes. It does not cover other behaviors like authentication or side effects, but the most critical safety trait is well disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, front-loaded with the primary action, and followed by a crucial safety note. The bilingual repetition is not wasteful, and every sentence serves a purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the core actions and the safety boundary, but it misses the purpose of the site parameter and any prerequisites or return behavior. Given the lack of annotations and output schema, more context would be beneficial for full completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already documents ids, and the description adds minimal context beyond that these are items to add to cart. However, the optional site parameter is entirely undocumented in the schema and unmentioned in the description, leaving a significant parameter semantic gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool adds items to cart and opens the review page, using specific verbs and resources. It also explicitly differentiates itself by noting it never clicks the final purchase button, which distinguishes it from purchase-completion tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage after chatbuy_search by referencing product IDs from that tool. It gives clear context for when to use it (for review before purchase) and imposes a safety boundary, but it does not explicitly name alternatives or exclusion criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.