Server Quality Checklist

Tool Scores

• get_account_summaryB
  3.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the agent knows this is a safe, repeatable read operation. The description adds the concept of 'aggregated' metrics, which provides some behavioral context beyond annotations, but it doesn't detail what aggregation entails or other traits like rate limits or auth needs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with no wasted words. It's front-loaded with the core purpose and appropriately sized for a tool with no parameters, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (simple read with no parameters), annotations cover safety and idempotency, but there's no output schema to explain return values. The description mentions 'aggregated portfolio metrics' but doesn't specify what metrics are included or the format, leaving gaps in completeness for agent usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% description coverage, so no parameter documentation is needed. The description doesn't add parameter details, which is appropriate here, earning a baseline score of 4 for not introducing unnecessary information.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool 'Get aggregated portfolio metrics' which specifies the action (get) and resource (portfolio metrics), but it's vague about what 'aggregated' means and doesn't distinguish it from sibling tools like get_portfolio or get_positions. It provides a basic purpose but lacks specificity and differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives such as get_portfolio or get_positions. The description implies usage for aggregated metrics but doesn't specify contexts, exclusions, or comparisons with siblings, leaving the agent without clear usage direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_order_historyB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the agent knows this is a safe, repeatable read operation. The description adds minimal behavioral context beyond this—it specifies that it retrieves 'filled orders' (implying historical data) and mentions filtering, but doesn't cover aspects like pagination, rate limits, or authentication needs. With annotations covering core safety, a baseline 3 is appropriate as the description adds some value but not rich behavioral details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's function and key parameters. It's front-loaded with the core purpose ('Get a list of filled orders') and includes essential filtering details without unnecessary elaboration. Every word earns its place, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (3 parameters, no output schema), the description is adequate but has gaps. It covers the basic purpose and filtering, but lacks details on output format, error handling, or integration with sibling tools. Annotations provide safety context, but without an output schema, the description could better prepare the agent for what to expect from the response. It's minimally viable but not fully comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with clear documentation for each parameter (begin, end, instId). The description adds marginal semantic value by framing 'instId' as an 'optional symbol filter' (though the schema marks it as required) and specifying 'date range', but doesn't provide additional syntax, format details, or constraints beyond what the schema already states. Baseline 3 is correct when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get a list of filled orders' with filtering by 'date range and optional symbol filter'. It specifies the verb ('Get') and resource ('filled orders'), making the intent unambiguous. However, it doesn't explicitly differentiate from sibling tools like 'get_open_orders' (which likely shows unfilled orders), leaving some room for improvement.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It mentions filtering capabilities but doesn't clarify scenarios where it's preferred over other tools like 'get_open_orders' or 'get_account_summary'. There's no mention of prerequisites, exclusions, or comparative use cases, leaving the agent to infer usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_open_ordersB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare this as read-only, idempotent, and non-destructive, so the description adds minimal behavioral context beyond stating it retrieves data. It doesn't disclose additional traits like response format, pagination, rate limits, or authentication needs. With annotations covering safety, a baseline 3 is appropriate as the description adds little value.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with no wasted words. It's front-loaded with the core action and resource, making it easy to parse. Every word earns its place by specifying 'all currently open orders' to clarify scope.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters, rich annotations (readOnlyHint, idempotentHint, destructiveHint), and no output schema, the description is adequate but minimal. It states what the tool does but doesn't explain return values or usage context. For a simple retrieval tool with good annotations, this is the minimum viable description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and schema description coverage is 100% (though empty). The description doesn't need to explain parameters, so it meets expectations. A score of 4 reflects that the description appropriately focuses on purpose rather than parameter details for a parameterless tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get') and resource ('open orders'), making the purpose immediately understandable. It distinguishes from siblings like 'get_order_history' by specifying 'currently open' rather than historical data. However, it doesn't explicitly contrast with other siblings like 'get_positions' or 'get_account_summary', preventing a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, timing considerations, or compare it to sibling tools like 'get_positions' or 'get_order_history'. The agent must infer usage context solely from the tool name and description.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_positionsB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate read-only, idempotent, and non-destructive behavior, so the description doesn't need to repeat this. It adds no additional behavioral context (e.g., rate limits, auth needs, or return format), but doesn't contradict annotations, resulting in a baseline score for minimal added value.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's function without any wasted words. It's appropriately sized and front-loaded, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema) and rich annotations, the description is adequate but minimal. It covers the basic purpose but lacks details on output format or when to use it, leaving some contextual gaps for the agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0 parameters and 100% schema description coverage, the schema fully documents the lack of inputs. The description implies no filtering or arguments are needed ('all derivative positions'), which aligns with the schema, earning a high baseline score for consistency.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get') and resource ('all derivative positions'), making the purpose understandable. However, it doesn't differentiate from sibling tools like 'get_portfolio' or 'get_account_summary', which might also retrieve position-related data, so it doesn't fully distinguish its specific scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention context, prerequisites, or comparisons to sibling tools like 'get_portfolio', leaving the agent without usage direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_portfolioA
  3.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, covering safety and idempotency. The description adds value by specifying the scope ('all assets') and that it returns a list, which isn't covered by annotations. No contradictions exist, and the description provides useful behavioral context beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose with zero wasted words. It directly communicates the tool's function without unnecessary elaboration, making it easy to parse and understand immediately.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 parameters, no output schema) and rich annotations, the description is adequate but has gaps. It doesn't explain the return format (e.g., structure of the asset list) or potential limitations (e.g., pagination, real-time data), which could be important for an agent despite the annotations covering safety aspects.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0 parameters and 100% schema description coverage, the schema fully documents the lack of inputs. The description adds semantic meaning by clarifying what is retrieved ('all assets in the account'), which compensates for the absence of parameters. This goes beyond the schema's structural information.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get') and resource ('list of all assets in the account'), making the purpose immediately understandable. It doesn't explicitly differentiate from sibling tools like get_positions, but the scope ('all assets') provides some implicit distinction. This is clear but lacks explicit sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like get_positions or get_account_summary. It doesn't mention prerequisites, context, or exclusions. The agent must infer usage from the tool name and description alone, which is insufficient for optimal tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.