Server Quality Checklist

Tool Scores

• get_account_infoD
  1.9/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. 'Get account information' implies a read operation but doesn't specify whether this requires authentication, what permissions are needed, whether it's rate-limited, what format the information is returned in, or if there are any side effects. For a tool with zero annotation coverage, this is a significant gap in behavioral transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  While the description is concise with just three words, it's under-specified rather than efficiently informative. 'Get account information' is too vague to be genuinely helpful—it doesn't earn its place by providing meaningful differentiation or context. True conciseness balances brevity with usefulness, which this description lacks.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of account information tools in a financial context (with multiple sibling tools for specific data), the description is incomplete. No annotations or output schema exist to compensate, and the description fails to clarify what this tool returns versus others like 'get_account_balance'. For a tool in a server with 10 sibling tools, this minimal description doesn't provide enough context for effective use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and schema description coverage is 100% (though with an empty schema). The description doesn't need to explain any parameters, which is appropriate. Since there are no parameters to document, the description's lack of parameter information doesn't detract from its adequacy in this dimension.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Get account information' is a tautology that essentially restates the tool name 'get_account_info'. While it indicates the verb 'get' and resource 'account information', it doesn't specify what kind of account information is retrieved or how this differs from sibling tools like 'get_account_balance' or 'get_positions'. The purpose is vague and lacks differentiation from related tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines1/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. With sibling tools like 'get_account_balance', 'get_positions', and 'get_snapshots' that likely retrieve specific subsets of account data, there's no indication of what makes this tool distinct or when it should be preferred over those more specific tools. No context, exclusions, or alternatives are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_contractsC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden but adds minimal behavioral context. It doesn't disclose whether this is a read-only operation, its performance characteristics (e.g., real-time vs. cached data), error handling, or output format. For a search tool with zero annotation coverage, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise ('Search for trading contracts')—a single sentence with no wasted words. It's front-loaded with the core action. However, this brevity borders on under-specification, as it omits necessary context for effective use.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (a search operation with 3 parameters) and lack of annotations and output schema, the description is incomplete. It doesn't explain what the search returns (e.g., list of contracts, metadata), how results are sorted or limited, or any behavioral traits. For a tool with no structured output, this leaves critical gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters (keyword, exchange, category) with clear descriptions. The description adds no additional meaning beyond what's in the schema, such as how parameters interact (e.g., AND/OR logic) or examples. Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the basic purpose ('Search for trading contracts') which is clear but vague. It specifies the resource ('trading contracts') but lacks detail about what kind of search this is (e.g., full-text, filtered, exact match) and doesn't differentiate from potential sibling tools like 'get_snapshots' or 'get_kbars' that might also retrieve contract-related data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The description doesn't mention prerequisites, context (e.g., during trading hours, for research), or exclusions. With siblings like 'get_snapshots' and 'get_kbars' that might overlap, this omission leaves the agent guessing about appropriate usage scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_kbarsC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states 'Get historical K-bar data,' implying a read-only operation, but doesn't cover critical aspects like rate limits, authentication needs, data freshness, or response format. For a data retrieval tool with zero annotation coverage, this leaves significant gaps in understanding how it behaves.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste. It's front-loaded with the core purpose and appropriately sized for the tool's complexity, making it easy to scan and understand quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (historical data retrieval with 4 parameters), lack of annotations, and no output schema, the description is incomplete. It doesn't explain what 'K-bar' entails, the data format returned, or any behavioral constraints. For a tool with these gaps, more context is needed to be fully helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with clear parameter definitions (e.g., 'Contract code,' 'Start date (YYYY-MM-DD)'). The description adds no additional meaning beyond the schema, such as explaining 'K-bar' context or parameter interactions. Baseline score of 3 is appropriate since the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'Get historical K-bar data' clearly indicates a retrieval operation for historical data, but it's vague about what 'K-bar' specifically refers to (likely candlestick data for financial instruments). It distinguishes from siblings like 'get_account_balance' or 'place_order' by focusing on market data rather than account or order operations, but doesn't explicitly differentiate from potential similar tools not listed.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. While siblings like 'search_contracts' might be related for finding contracts, there's no mention of prerequisites, timing, or complementary tools. The description implies usage for historical data analysis but lacks explicit context or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_ordersC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. 'List all orders' implies a read-only operation, but it doesn't specify if this requires authentication, what the return format is (e.g., list, paginated), or any rate limits. It misses key details like whether it returns real-time or historical data, making it inadequate for a tool with zero annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise with just three words, which is efficient for a simple tool. It's front-loaded with the core action and resource. However, it might be overly terse, lacking necessary context that could make it more helpful without adding significant length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (a read operation with no parameters) and no annotations or output schema, the description is incomplete. It doesn't explain what 'orders' entails (e.g., type, status), the return format, or any behavioral aspects like authentication needs. For a tool in a financial context with siblings like 'cancel_order', more detail is warranted to ensure correct usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so no parameters need documentation. The description doesn't add parameter details, but since there are no parameters, this is acceptable. A baseline of 4 is appropriate as the schema fully covers the lack of parameters, and the description doesn't need to compensate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'List all orders' clearly states the verb ('List') and resource ('orders'), making the purpose understandable. However, it doesn't differentiate from sibling tools like 'get_positions' or 'get_snapshots' which might also retrieve order-related data, and the scope 'all orders' could be ambiguous without context about filtering or pagination.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. For example, it doesn't specify if this is for retrieving historical orders, open orders, or all orders, nor does it mention prerequisites like authentication or rate limits. The description lacks any context to help an agent choose between this and siblings like 'get_positions' or 'search_contracts'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cancel_orderC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It states the tool cancels orders and mentions an environment variable requirement, but doesn't describe what cancellation entails (immediate vs. pending, reversibility, confirmation response), potential side effects, or error conditions. This leaves significant gaps for a mutation tool in a trading context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that communicates the core purpose and a key requirement. There's no unnecessary elaboration, though it could be slightly more structured by separating the purpose from the prerequisite.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a mutation tool with no annotations and no output schema, the description is insufficient. It doesn't explain what happens after cancellation (success/failure responses, order status changes), doesn't address potential risks or limitations, and provides minimal behavioral context. The environment variable mention is helpful but doesn't compensate for the broader gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the single parameter 'order_id' clearly documented in the schema. The description doesn't add any parameter-specific information beyond what the schema provides, such as format examples or where to obtain order IDs. Given the high schema coverage, the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('cancel') and target ('an existing order'), providing a specific verb+resource combination. However, it doesn't differentiate this tool from potential alternatives like modifying or deleting orders, which aren't present in the sibling list but could be relevant in a trading context.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions a prerequisite ('requires SHIOAJI_TRADING_ENABLED=true'), which provides some usage context, but offers no guidance on when to use this tool versus alternatives like modifying orders or letting orders expire. It doesn't explain relationships with sibling tools like list_orders (to find order_id) or place_order (the opposite action).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_account_balanceC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. While it implies a read operation ('Get'), it doesn't specify whether this requires authentication, has rate limits, returns real-time or cached data, or what format the balance information takes. For a financial data tool with zero annotation coverage, this leaves significant gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that gets straight to the point without unnecessary words. It could be slightly improved by front-loading more critical information about behavioral aspects, but it's appropriately sized for a simple query tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a financial balance query tool with no annotations and no output schema, the description is insufficient. It doesn't explain what 'balance information' includes (cash, margin, unrealized P/L?), whether it's real-time, authentication requirements, or error conditions. The agent would be operating with significant uncertainty about the tool's behavior and output.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 100% description coverage with a well-documented enum parameter, so the description doesn't need to compensate. The description mentions 'stock and/or futures accounts' which aligns with the enum values but doesn't add meaningful semantics beyond what the schema already provides. Baseline 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Get') and resource ('account balance information'), and specifies the account types covered ('stock and/or futures accounts'). However, it doesn't differentiate from sibling tools like 'get_account_info' which might provide overlapping or related information.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'get_account_info' or 'get_positions'. It doesn't mention prerequisites, timing considerations, or any context that would help an agent choose between this and other account-related tools in the server.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_positionsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool retrieves data ('Get'), implying a read-only operation, but doesn't specify permissions, rate limits, data freshness, or response format. For a tool with zero annotation coverage, this is a significant gap in behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that clearly states the tool's purpose without unnecessary words. It is appropriately sized and front-loaded, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete. It doesn't explain what data is returned (e.g., position details, quantities, values), how results are structured, or any behavioral traits like error handling. For a data retrieval tool with no structured output information, this leaves significant gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with a single parameter 'account_type' fully documented (including enum values and default). The description mentions 'stock and/or futures accounts', which aligns with the schema but adds no additional semantic context beyond what the schema already provides. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get current positions from stock and/or futures accounts'. It specifies the verb ('Get'), resource ('positions'), and scope ('stock and/or futures accounts'), making it easy to understand what the tool does. However, it doesn't explicitly differentiate from sibling tools like 'get_account_balance' or 'get_account_info', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like 'get_account_balance' or 'get_account_info', nor does it specify prerequisites or contexts where this tool is preferred. This leaves the agent without clear usage instructions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_snapshotsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions 'real-time' market snapshots, which hints at live data, but doesn't cover critical aspects like rate limits, authentication needs, data freshness, or error handling. This is inadequate for a tool that likely interacts with market data.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste. It's front-loaded with the core action and resource, making it easy to parse quickly without unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of market data tools and the lack of annotations and output schema, the description is incomplete. It doesn't explain what a 'snapshot' includes (e.g., prices, volumes), how results are structured, or any behavioral traits, leaving significant gaps for the agent to operate effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 100%, with the 'contracts' parameter well-documented as a list of contract codes. The description doesn't add any meaning beyond this, such as explaining contract code formats or examples, but the schema provides sufficient baseline information.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Get') and resource ('real-time market snapshots'), making the purpose immediately understandable. However, it doesn't differentiate from siblings like 'get_kbars' (which likely provides historical data) or 'search_contracts' (which might find contracts rather than their snapshots), missing full sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. With siblings like 'get_kbars' (possibly for historical data) and 'search_contracts' (for finding contracts), there's no indication of context, prerequisites, or exclusions, leaving the agent to infer usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_terms_statusB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but offers minimal behavioral insight. It implies a read-only operation ('check') but doesn't disclose authentication needs, rate limits, response format, or whether it's a simple status check versus a detailed report. This is inadequate for a tool with zero annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with no wasted words. It's front-loaded with the core purpose and structured clearly, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the lack of annotations and output schema, the description is incomplete. It doesn't explain what the check returns (e.g., boolean status, detailed report), how results are formatted, or error conditions, leaving significant gaps for agent understanding in a context with no structured support.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters with 100% schema description coverage, so the baseline is 4. The description doesn't add parameter details, but that's unnecessary since no parameters exist, making it appropriately minimal in this dimension.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: checking both service terms signing status and API testing completion. It uses specific verbs ('check') and identifies the resources (status and completion), but doesn't differentiate from sibling tools like 'run_api_test' which might overlap with API testing aspects.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives. The description doesn't mention prerequisites, timing considerations, or relationships to sibling tools like 'run_api_test' or 'get_account_info', leaving the agent without context for selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_api_testB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It mentions the tool runs 'login and order tests', implying it performs actions (potentially mutations), but doesn't disclose behavioral traits like whether it's safe (read-only vs. destructive), requires specific permissions, has side effects (e.g., creating test data), or includes rate limits. This leaves significant gaps for a tool that likely interacts with systems.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero waste: 'Run API test for service terms compliance (login and order tests)'. It is front-loaded with the core action and includes clarifying details in parentheses, making it appropriately sized for a no-parameter tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (likely a test tool with potential system interactions), lack of annotations, and no output schema, the description is incomplete. It doesn't explain what the test does (e.g., simulates actions, validates responses), what it returns (e.g., pass/fail results, logs), or any constraints (e.g., environment-specific). For a tool with no structured safety or output info, more detail is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so no parameter documentation is needed. The description adds value by implying the test scope ('login and order tests'), which provides context beyond the empty schema. However, it doesn't detail any optional configurations or inputs, though none are required.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Run API test for service terms compliance (login and order tests)'. It specifies the verb ('Run'), resource ('API test'), and scope ('service terms compliance'), distinguishing it from sibling tools like check_terms_status or place_order. However, it doesn't explicitly differentiate from all siblings (e.g., how it differs from check_terms_status beyond being a test).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., whether authentication is needed), timing (e.g., for compliance verification), or exclusions (e.g., not for production use). Sibling tools like check_terms_status might overlap in purpose, but no comparison is made.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• place_orderB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions an environment variable requirement, which is useful context, but fails to describe critical traits such as whether this is a destructive/mutative operation (likely yes, given 'place'), what permissions or authentication are needed, potential rate limits, or the expected response format. This leaves significant gaps for an agent to understand the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose and includes a necessary prerequisite. There is no wasted text, and it's appropriately sized for the tool's complexity, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (a trading order placement with 5 parameters, no output schema, and no annotations), the description is incomplete. It lacks details on behavioral aspects like mutation effects, error handling, or return values, and doesn't fully guide usage relative to siblings. While concise, it doesn't provide enough context for an agent to confidently invoke this tool in a trading environment.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds no additional parameter information beyond what's in the schema, such as clarifying 'contract' format or 'order_type' meanings. This meets the baseline of 3, as the schema handles the heavy lifting, but the description doesn't compensate or enhance understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'place' and the resource 'trading order', making the purpose specific and understandable. However, it doesn't explicitly distinguish this tool from sibling tools like 'cancel_order' or 'list_orders', which would require mentioning it's for creating new orders rather than modifying or viewing existing ones.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a prerequisite ('requires SHIOAJI_TRADING_ENABLED=true'), which gives some context for when to use it. However, it doesn't offer explicit guidance on when to choose this tool over alternatives like 'cancel_order' or how it relates to other trading tools, leaving usage somewhat implied rather than fully clarified.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.