Server Quality Checklist

Tool Scores

• cancel_orderA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Given no annotations, the description reveals the refund behavior ('unfulfilled remainder is refunded to the key's USD balance'). This adds context beyond the input schema, though it omits details like irreversibility or error conditions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short, focused sentences with no filler. The critical information (cancellation and refund) is front-loaded, and the parameter usage is clearly stated.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has low complexity and no output schema, but the description fails to mention constraints like whether fully fulfilled orders can be cancelled or if partial cancellations are supported. The refund detail is helpful but leaves gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter 'orders' has 100% schema coverage with description. The description adds 'Pass orders as a list of order IDs', which reiterates the schema. No additional semantic information is provided beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool cancels orders and refunds the unfulfilled remainder, using a specific verb ('Cancel') and resource ('orders'). It distinguishes from siblings like 'create_order' and 'order_status' by focusing on cancellation and refund.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not explicitly state when or when not to use the tool, nor does it compare with sibling tools. It implies usage via the verb 'cancel' but lacks guidance on scenarios like already fulfilled orders.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• order_statusA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description must carry the full burden. It implicitly indicates a read operation ('Get the status') and lists returned fields, but it does not explicitly state that the tool is read-only, idempotent, or free of side effects. This is adequate but could be more transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, each serving a clear purpose: defining the function and detailing parameters and return values. No redundant words; every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 params, no output schema, no nested objects), the description covers core functionality well. It could benefit from explaining the meaning of the return fields (charge, start_count, status, remains), but the context is sufficient for basic usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds value by explaining that 'orders' accepts up to 100 IDs and that either 'order' or 'orders' can be used. This extra context goes beyond the schema's basic descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Get', the resource 'status of one or more orders', and specifies how to pass orders (single or list). It lists the returned fields, making the purpose unambiguous and distinct from sibling tools like cancel_order or create_order.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for retrieving order status but does not explicitly state when to use it over alternatives. No guidance on when not to use or context for distinguishing from siblings, though the purpose is clear enough for basic use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_balanceA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It correctly indicates a read operation (get balance), but does not disclose additional behavioral traits such as authentication requirements, rate limits, or whether the balance is real-time.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single 11-word sentence that is perfectly concise and front-loaded with the key action 'Get the remaining USD balance'. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no parameters or output schema, the description is largely sufficient. It could mention whether the balance is cached or requires authentication, but for a simple query it is complete enough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the description adds no parameter meaning beyond what the schema provides. Baseline score of 4 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it retrieves the remaining USD balance of the configured reseller API key, specifying the resource (balance), currency (USD), and scope (configured reseller API key). It distinguishes effectively from sibling tools that deal with orders and services.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use for checking available funds, but does not explicitly state when to use versus alternatives (e.g., before creating an order). It provides clear context for its intended use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_servicesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It accurately describes a read-like listing operation with live rates and no API key requirement. While it doesn't explicitly state 'read-only', the context implies no destructive behavior. The description adds useful behavioral context beyond the empty schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no wasted words. The first sentence states the purpose and key data points; the second provides usage context. Information is front-loaded and efficiently structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters, no output schema, and the presence of sibling tools like 'create_order' and 'get_balance,' the description is complete. It explains the tool's role in the workflow (discovery before ordering) and all necessary details for an AI agent to use it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are zero parameters, and schema coverage is 100% trivially. The description adds meaning by detailing what the output contains (rates per 1000, min/max, link requirement), compensating for the lack of a structured output schema. It clarifies the semantic value of the tool beyond the empty input schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') and resource ('Stars Cash Flow service catalog'), and explicitly mentions the key attributes (live USD rates, min/max quantity, link requirement). It clearly distinguishes itself from sibling tools like 'create_order' and 'get_balance' by positioning itself as a discovery tool for pricing orders.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use this to discover service IDs and price an order before calling create_order,' providing clear when-to-use guidance. It also states 'No API key required,' a prerequisite. It does not mention when not to use it or alternatives, but given zero parameters and a straightforward purpose, this is adequate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_orderA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Highlights key behavioral trait 'THIS SPENDS the key's USD balance'. Explains two-step idempotency pattern. No annotations provided, but lacks details on auth or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, front-loaded with purpose and key warning. No filler, efficient and direct.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Explains two-step process, spends balance, required fields. No output schema, but missing return value description is acceptable. Adequate for use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage 100%, but description adds context: explains confirm as two-step toggle and link as 'target channel/post/bot link', beyond schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Title explicitly states 'spends balance', description specifies 'Stars Cash Flow order' and distinguishes from siblings (cancel, balance, list, status). Clear verb+resource.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit two-step design: call without confirm for estimate, then with confirm=true to place order. No alternatives among siblings, but usage pattern is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.