Server Quality Checklist

Tool Scores

• market_snapshotB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the disclosure burden. 'Read' and 'mock' convey that this is a non-destructive, non-production data operation, which is useful. However, it does not explain normalization rules, whether the quote is current/last available, error behavior, or any other operational trait, and no annotations compensate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no filler; every word contributes. It is appropriately short for a one-parameter read operation, though terse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a one-parameter read tool with an output schema, the core call shape is clear, but there is no usage guidance, no caveats, and no explanation of the 'normalized' aspect. The missing usage context and parameter semantics leave the agent to guess when and how to invoke this over alternatives.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has one required parameter, 'symbol', with 0% description coverage, and the description adds no explicit parameter explanation. The phrase 'market-data quote' only weakly implies that symbol identifies the instrument. A format or example would be needed to reach the low-coverage compensation bar.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific verb ('Read') and a specific resource ('normalized mock market-data quote'), so an agent can tell this is the market-data retrieval tool. Among siblings (generate_alpha_signal, validate_order, submit_paper_order, trip_risk_circuit_breaker), it uniquely reads a quote rather than generating, validating, submitting, or tripping something.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives no explicit condition for use, no exclusions, and no pointer to an alternative sibling. It only states what the tool does; the agent must infer when to pick it. No prerequisites or context are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• submit_paper_orderA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry behavioral weight. It discloses that the order is paper-only and that prior risk validation is mandatory, which are useful traits. However, it does not explain failure behavior, required permissions, or side effects, so it only partially satisfies the burden.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no filler. Every part contributes meaning: 'submit', 'paper-only order', and the pre-trade validation requirement.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having an output schema, the tool is missing essential usage context: parameter semantics are completely undocumented and there are no annotations. The description provides a high-level workflow hint but is not sufficient for an agent to construct valid calls reliably.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description provides no meaning for any of the five required parameters. It does not compensate for the schema's lack of documentation, leaving the agent to guess valid values for side, quantity, limit_price, and client_order_id.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('submit') and resource ('paper-only order'), and adds a key ordering constraint ('after mandatory pre-trade risk validation') that distinguishes it from sibling validation tools. This is a clear, non-tautological purpose statement.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'after mandatory pre-trade risk validation' gives the agent explicit context about when to invoke this tool relative to the validation workflow. It does not name alternatives or state when not to use it, but the precondition is clear enough for correct sequencing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_orderA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral transparency burden. It does add meaningful traits: 'deterministic' and 'without submitting an order,' which rule out randomness and order placement side effects. It doesn't disclose input requirements or failure behavior, but the core safety property is stated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no filler. It front-loads the action and the key safety distinction, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  A tool with five required parameters, no annotations, and 0% schema coverage needs more than a one-line description for correct invocation. While an output schema exists, it does not compensate for missing parameter semantics and unclear boundaries versus sibling risk tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not explain any of the five required parameters. The agent is left with only parameter names and no guidance on formats, allowed values, or meaning, so the description adds no parameter-level value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Run') with a clear resource ('pre-trade risk checks') and adds the key distinction 'without submitting an order.' This makes it easy to distinguish from order-submission siblings like submit_paper_order.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'without submitting an order' gives a clear context for when to use this tool: when validation is needed but execution is not. However, it does not explicitly name alternatives or state when not to use it, especially relative to trip_risk_circuit_breaker.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_alpha_signalA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full behavioral burden. It discloses two key traits: the model is a mock and the call has no order side effects. However, it omits details about return shape, errors, or feature-vector requirements, so disclosure is partial.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences with no filler. The core action and output are front-loaded, and the crucial safety clarification 'never places an order' is included without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter mock tool, the purpose and side-effect profile are clear, and an output schema exists so return-value details are not required. The only notable gap is the unexplained 'features' parameter, but overall the description is sufficiently complete for the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description adds no meaning for either parameter. 'symbol' is weakly inferable from the name, but 'features' is entirely unexplained—no length, ordering, or normalization context. The description fails to compensate for the schema gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific action ('Run'), the resource ('mock PyTorch GNN'), and the output ('alpha score'). It also explicitly distinguishes itself from order-placing siblings with 'never places an order', making its purpose unmistakable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'Informational only; never places an order' gives a clear context and an explicit when-not-to-use boundary. It does not name alternative siblings like market_snapshot or specify when to prefer this one, but the informational intent is clearly conveyed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• trip_risk_circuit_breakerA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden and it discloses three key traits: irreversibility, scope ('in this process'), and what it affects ('new order attempts'). It does not cover operational details like idempotency or whether pending orders are affected, hence not a 5.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One tightly written sentence with no filler; the core action and trigger are front-loaded. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a one-parameter tool with an output schema, the description covers the essential what, scope, and when. The 'reason' parameter semantics and a bit more operational context are absent, but the tool is simple enough that the missing detail is minor.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has one required string 'reason' with 0% description coverage, and the description adds no parameter-level meaning. The name alone gives some clue, but the description was required to compensate for the schema gap and does not.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb ('block'), resource ('new order attempts'), and a critical modifier ('irreversibly'), plus the emergency context ('incident or anomaly'). This is clearly distinct from sibling tools like submit_paper_order or validate_order.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Gives a clear trigger condition: 'use for an incident or anomaly.' It does not explicitly name when-not-to-use or alternatives, but the emergency context is sufficient to guide selection against the normal-flow siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.