Server Quality Checklist

Tool Scores

• close_positionC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It states the action is to 'close' a position, implying a destructive/mutative operation, but doesn't disclose critical behavioral traits: whether this requires specific permissions, if it's irreversible, what happens on failure (e.g., symbol not found), rate limits, or confirmation details. For a mutation tool with zero annotation coverage, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded: the first sentence states the core purpose clearly. The Args and Returns sections are structured but could be more integrated. No wasted sentences, though it could be slightly more polished (e.g., merging sections).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given complexity (a destructive financial operation), no annotations, no output schema, and low schema coverage, the description is incomplete. It lacks details on behavioral risks, error handling, return format (beyond vague 'Confirmation'), and context like account or position type. For such a tool, more comprehensive guidance is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds meaning by specifying 'symbol' as a 'Stock symbol to close position for', which clarifies the parameter's purpose beyond the schema's generic 'Symbol' title. However, it doesn't provide format details (e.g., ticker format like 'AAPL'), validation rules, or examples, leaving some ambiguity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Close') and target resource ('an open position for a specific symbol'), which is specific and unambiguous. It distinguishes from siblings like cancel_order (which cancels unfilled orders) and place_*_order tools (which open positions). However, it doesn't explicitly mention what type of position (long/short) or account context, leaving slight room for improvement.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., must have an open position for the symbol), exclusions (e.g., cannot close if no position exists), or comparisons to sibling tools like cancel_order (for orders) or place_*_order (for opening). Usage is implied but not explicitly stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• place_market_orderB
  3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but only states the basic action. It doesn't disclose critical behavioral traits like authentication requirements, rate limits, whether this is a real trade or simulation, what happens on execution failure, or confirmation details format. For a financial transaction tool, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured with a clear purpose statement followed by parameter and return sections. Every sentence adds value, though the return statement could be more specific. It's appropriately sized for a 3-parameter tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a financial trading tool with no annotations and no output schema, the description is incomplete. It doesn't address critical context like authentication, execution guarantees, error conditions, or what 'order confirmation details' includes. The agent lacks sufficient information to use this tool safely and effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description provides clear semantic meaning for all three parameters beyond the schema's 0% coverage. It explains symbol format with an example, clarifies quantity can be fractional, and specifies side options. This compensates well for the schema's lack of descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Place a market order') and resource ('stock'), distinguishing it from sibling tools like place_limit_order. However, it doesn't explicitly differentiate market orders from other order types beyond the name, missing an opportunity to clarify the immediate execution characteristic.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like place_limit_order or place_stop_order. The description lacks context about market conditions, urgency, or price sensitivity that would help an agent choose between order types.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cancel_orderB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but offers minimal behavioral context. It states the action is a cancellation but doesn't disclose critical traits like permission requirements, whether cancellation is reversible, rate limits, or what happens on failure (e.g., if order is already filled).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is highly concise and well-structured: a clear purpose statement followed by brief 'Args' and 'Returns' sections. Every sentence earns its place with no wasted words, and key information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (a mutation with no annotations, 0% schema coverage, and no output schema), the description is incomplete. It lacks details on behavioral constraints, error conditions, return value specifics beyond 'confirmation', and how it differs from sibling tools in practice.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds meaningful context by explaining 'order_id' as 'ID of the order to cancel', clarifying its role beyond the schema's basic string type. However, it doesn't specify format (e.g., numeric, UUID) or sourcing details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Cancel') and resource ('an open order by its ID'), making the purpose unambiguous. It distinguishes from siblings like 'close_position' by specifying it cancels orders rather than positions, though it doesn't explicitly contrast with all order-placement tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't specify prerequisites (e.g., order must be open), exclusions (e.g., cannot cancel filled orders), or compare with sibling tools like 'close_position' for similar actions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_portfolio_summaryB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool returns a summary but doesn't describe what 'comprehensive' entails, whether it's real-time or cached data, if there are rate limits, authentication requirements, or error conditions. For a financial tool with zero annotation coverage, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately concise with two sentences: one stating the purpose and one describing the return. Both sentences earn their place by providing essential information. It could be slightly more structured by front-loading the most critical information, but it's efficiently written without unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (financial portfolio summary), lack of annotations, and no output schema, the description is minimally adequate. It explains what the tool does and what it returns at a high level, but doesn't provide details about the return structure, data freshness, or error handling. For a tool in a financial context with siblings that perform transactions, more completeness would be expected.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters with 100% schema description coverage, so the schema fully documents the absence of parameters. The description appropriately doesn't discuss parameters since none exist. It earns a 4 because it doesn't waste space on parameter discussion when none are needed, though it doesn't add value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get a comprehensive summary of the portfolio including account details and open positions.' It specifies the verb ('Get') and resource ('portfolio summary'), and distinguishes it from siblings like get_account_info_tool by mentioning both account details AND positions. However, it doesn't explicitly contrast with get_account_info_tool, so it's not a perfect 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention when to choose get_portfolio_summary over get_account_info_tool (which might provide only account info) or when it's appropriate to use this summary tool versus individual position/order tools. There's no context about prerequisites or timing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_account_info_toolB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool returns account summary with balance and status, but doesn't cover critical aspects like whether this is a read-only operation, authentication requirements, rate limits, or error handling. For a tool with zero annotation coverage, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and well-structured: it states the purpose in one clear sentence and the return value in another. Every sentence earns its place, with no redundant or unnecessary information, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema, no annotations), the description is adequate but has clear gaps. It explains what the tool does and what it returns, but lacks behavioral context (e.g., safety, auth) and usage guidelines. For a basic read operation, this is minimally viable but could be more informative.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and the schema description coverage is 100%, so there's no need for parameter details in the description. The baseline for this scenario is 4, as the description appropriately doesn't waste space on non-existent parameters and focuses on the tool's purpose and return value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get current account information.' It specifies the verb ('Get') and resource ('current account information'), making it easy to understand what the tool does. However, it doesn't explicitly differentiate from sibling tools like 'get_portfolio_summary', which might provide overlapping functionality.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention any prerequisites, context for usage, or comparisons with sibling tools such as 'get_portfolio_summary'. Without this, an agent might struggle to choose between similar tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• place_limit_orderB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. While it implies a write operation ('Place'), it doesn't mention critical aspects like authentication requirements, rate limits, execution guarantees, or potential side effects (e.g., funds reservation). This is inadequate for a financial transaction tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (Args, Returns) and uses bullet-like formatting. Every sentence adds value, though the 'Returns' section could be more specific. It's appropriately sized for a 5-parameter tool without unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a financial order placement tool with no annotations and no output schema, the description is incomplete. It lacks critical context about authentication, error conditions, order status tracking, and what 'Order confirmation details' actually contains. The agent would need to guess important behavioral aspects.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Given 0% schema description coverage, the description fully compensates by explaining all 5 parameters with clear semantics. It defines each parameter's purpose, provides examples (e.g., 'AAPL'), clarifies numeric types (e.g., 'can be fractional'), enumerates valid values (e.g., 'buy' or 'sell'), and explains constraints (e.g., 'Maximum price for buy').

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Place a limit order') and resource ('stock'), distinguishing it from sibling tools like place_market_order or place_stop_order. It precisely defines the tool's function without being tautological.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like place_market_order or place_stop_limit_order. It lacks context about appropriate scenarios, prerequisites, or exclusions, leaving the agent to infer usage from the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• place_stop_orderA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It mentions the tool 'places' an order (implying a write/mutation) but doesn't disclose behavioral traits like whether it requires authentication, has rate limits, what happens on execution failure, or confirmation details format. The description adds minimal behavioral context beyond the basic action.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and well-structured with clear sections for purpose, args, and returns. Each sentence earns its place, though the 'Returns' section is vague ('Order confirmation details') and could be more specific.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (financial order placement with 5 parameters), no annotations, and no output schema, the description is moderately complete. It covers parameters well but lacks behavioral context (e.g., auth, errors) and output details, leaving gaps for a mutation tool in a financial context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description fully compensates by explaining all 5 parameters in detail: symbol (with an example), quantity (including fractional shares), side (with valid values), stop_price (its role), and time_in_force (with options and default). It adds meaning beyond the bare schema titles.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Place a stop order'), the resource ('to buy or sell a stock'), and the condition ('when it reaches a specified price'). It distinguishes this tool from siblings like place_limit_order and place_market_order by specifying it's a stop order triggered by price.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context through the tool name and description, suggesting it's for placing stop orders. However, it doesn't explicitly state when to use this versus alternatives like place_stop_limit_order or place_limit_order, nor does it mention prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• place_stop_limit_orderA
  4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool places an order but doesn't mention critical behaviors like execution risks, confirmation details, error handling, or prerequisites (e.g., account permissions). For a financial trading tool with potential monetary impact, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the core purpose, followed by organized sections for Args and Returns. Every sentence adds value without redundancy, making it efficient and easy to parse for an AI agent.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a financial order tool with no annotations and no output schema, the description covers parameters well but lacks behavioral context (e.g., execution details, errors) and return value specifics. It's adequate for basic usage but incomplete for safe and informed operation in a trading environment.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate fully. It provides clear semantics for all 6 parameters (e.g., 'symbol: Stock symbol', 'stop_price: Price that triggers the order'), adding essential meaning beyond the bare schema. This effectively documents each parameter's purpose and usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Place a stop-limit order') and distinguishes it from siblings by mentioning it 'combines stop and limit order features.' This directly differentiates it from place_stop_order and place_limit_order, making the purpose explicit and distinct.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by explaining it combines features of stop and limit orders, which helps differentiate it from siblings like place_market_order. However, it lacks explicit guidance on when to use this tool versus alternatives (e.g., no mention of specific market conditions or strategies), so it doesn't fully reach the highest score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.