Server Quality Checklist

Tool Scores

• get_positionsA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It explains that negative quantity means short, and notes market value is included 'when the broker prices it', acknowledging variability. However, it does not mention whether this is a read-only operation, any error conditions, or pagination, which would be expected without annotation support.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that front-loads the core action and then adds relevant field details and the parameter. Every part earns its place; there is no waste or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one optional parameter and no output schema, the description adequately covers the purpose, parameter, and key output semantics. It does not explicitly state that the result is a list or how errors are handled, but these are minor given the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already covers the single optional 'broker' parameter with a description and example. The description only reinforces its optionality ('Optionally filter by broker id') without adding extra meaning, so it remains at the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves open positions across all accounts, specifying the fields returned (symbol, quantity, market value) and the meaning of negative quantity. This distinguishes it from siblings like get_trades (trade history) and get_stats (statistics) without ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use it (for positions) but provides no explicit guidance on when not to use it or alternatives. It does not mention sibling tools or situations where another tool would be more appropriate, leaving the agent to infer based on the action and resource.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refreshA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description carries the entire burden. It discloses the core behavioral trait (bypassing cache and re-fetching all brokers) and the return format (per-broker success/failure). However, it does not mention side effects such as potential latency, network calls, rate-limit impact, or whether it is read-only. This is a moderate disclosure but leaves out important operational implications for an agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that front-loads the primary action, specifies the scope, and immediately states the output. Every word adds value—'Bypass the cache' sets the key behavior, 're-fetch every configured broker right now' defines scope and urgency, and 'Returns per-broker success/failure' completes the contract. There is no waste or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with no parameters and no output schema, the description is quite complete. It tells the agent what the tool does, what it operates on, and what it returns. The only missing details are minor—like whether the operation is asynchronous or may take a long time, or whether it can trigger error rates—but these are not critical for correct invocation. The description provides enough context for an agent to use it appropriately.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline is 4. The description adds no parameter-specific info (as there are none), but it does enrich the overall meaning by clarifying the action and return structure. The schema is trivially covered since no parameters exist, and the description fully compensates for the lack of parameters by explaining the operation’s effect and output.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Bypass the cache and re-fetch every configured broker right now') and a clear resource scope ('every configured broker'). It also mentions the return type (per-broker success/failure), which distinguishes it from sibling tools like list_brokers or get_stats that focus on listing or retrieving specific data. The purpose is unambiguous and immediately differentiates this tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use it (when you want to bypass cache and get fresh data immediately) but does not explicitly contrast with alternatives or state when not to use it. It lacks guidance like 'use this instead of list_brokers when you need current data' or 'avoid if you don't need real-time data.' The phrase 'right now' conveys urgency, but the selection criteria are not spelled out.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_statsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of disclosing behavior. It discloses key traits: computation is FIFO-matched, amounts are in native currency, and mixed-currency totals are approximate. It also implicitly indicates a read-only nature. However, it does not explicitly state read-only status, mention any rate limits, or clarify whether the 'top positions' are sorted by value or quantity. The currency and approximation notes add valuable transparency beyond a simple 'get stats'.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, information-dense sentence that front-loads the core purpose and then specifies the exact metrics. The trailing note about currency handling is relevant and concise. There is no filler or repetition. Every clause adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters and no output schema, the description provides a solid overview of what the tool returns. It lists all major categories of computed stats and clarifies the currency handling. However, it could be slightly more explicit about the return format (e.g., whether it's a single object or a list) and whether any filtering is possible. For a stateless aggregation tool, this is nearly complete, but not perfectly exhaustive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline is 4. The description does not need to add parameter semantics because there are none. The schema coverage is 100% (vacuously), and the description complements the schema by explaining what the output covers. No further parameter information is required.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the resource ('performance stats across the whole portfolio') and enumerates the specific aggregates returned (total equity, equity by broker, top positions, FIFO-matched trade stats, per-symbol breakdown). This clearly distinguishes it from siblings like get_positions and get_trades, which focus on raw data rather than computed summary metrics.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  While the description makes the tool's purpose clear, it does not explicitly state when to use this tool versus alternatives, nor does it mention any exclusions. The usage context is implied as 'for aggregated portfolio stats', but there is no direct guidance on when get_stats should be preferred over get_positions or get_trades. Since the tool takes no parameters, the lack of alternatives guidance is a minor gap.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_tradesA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral burden and does a good job: it discloses the broker-dependent lookback window ('most recent window each broker exposes'), ordering ('newest first'), and all-account scope. It does not describe response shape or pagination, but these are less critical for a read-only history query.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence front-loads the core purpose, then appends temporal scope, filters, and ordering. There is no filler or redundant restatement of the schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with three optional, fully documented parameters and no output schema, the description covers the essential invocation facts: what is returned, scope, temporal limitation, filters, and sort order. It could mention trade record fields, but the absence is not a blocker for selecting and calling the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema already documents limit, broker, and symbol. The description adds only 'and/or' filter flexibility and 'newest first' ordering, which is minimal extra value over the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names the resource ('trade history'), scope ('across all accounts'), and key characteristics ('most recent window each broker exposes', 'newest first'). This clearly distinguishes it from sibling tools like get_positions (positions vs history) and list_brokers/list_accounts (metadata vs transactions).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives useful usage context: the tool is for recent historical trades across all accounts, with optional broker and symbol filters. It doesn't explicitly name alternatives or exclusions, but the scope and filters make when to call it reasonably clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_brokersA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly states that values are never shown, only whether variables are set, which is a key security-relevant behavior. It also implies a read-only check of the server's configuration, though it does not partially ignore the read-only nature explicitly—still, the transparency about not revealing secrets is valuable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no wasted words. It front-loads the primary purpose (list brokers), then adds a crucial caveat about not showing values. Perfectly concise and clear.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter list tool with no output schema, the description adequately conveys what is returned: the list of brokers, their env var names, and configuration status. It omits specific output formatting (e.g., list vs table) but that is minor given the simplicity and the clear security note about not exposing values. Overall, it is complete enough for an agent to call correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the schema covers 100% of them (i.e., an empty schema). The description correctly adds no parameter details since there are none. Per calibration, 0 params baseline is 4, and the description is consistent.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool lists every supported broker, the environment variables for its credentials, and whether each is configured. The verb 'list' and resource 'brokers' are clear, and the function is distinct from siblings like list_accounts or get_stats, which focus on other data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description conveys the tool's purpose clearly but does not explicitly state when to use it relative to alternatives or when not to use it. Since it is a zero-parameter list operation with no obvious exclusions, the context is implied but lacks explicit guidance for selection among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_accountsA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that the tool uses a short-lived cache, meaning data may be stale, and instructs to call refresh for live numbers, which is a behavioral trait not inferable from the name alone. It also lists the return fields.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loading the purpose and return fields before explaining the caching behavior. Every clause earns its place with no filler words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (0 params, no output schema), the description covers all necessary context: what it returns (id, name, broker, currency, equity, cash when reported), the caching behavior, and the alternative for live data. Nothing needed to invoke the tool correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  This tool has zero parameters, so the schema fully documents them (vacuously) with 100% coverage. The description adds no parameter-level information, but none is needed; the baseline for zero-parameter tools is 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('List') and a clear resource ('all connected accounts across every configured broker') along with the exact fields returned. This clearly differentiates it from sibling tools like list_brokers, which would list brokers not accounts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly directs the agent to call refresh for live numbers, providing an alternative for the specific condition of needing up-to-date data. This is a clear when-to-use alternative, matching the high-quality pattern of naming the alternative and the condition.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.