Server Quality Checklist

Tool Scores

• create_researchA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses that it waits up to 300s, may time out, and returns a jobId in error for checking status. It also mentions the action is saving a report (creating a resource), but lacks explicit statements about mutation or side effects. However, with no annotations provided, the description carries the burden and does a good job by noting the wait and jobId behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single paragraph but richly detailed, front-loaded with the core purpose. It is somewhat long, but every sentence adds value, covering usage, parameters, return values, and follow-up actions. Could be slightly more structured (e.g., bullets) but still efficient and well-organized.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has one param, no output schema, and no annotations, so the description must cover return values and error behavior. It does: returns analysisId, title, symbol, overallScore, overallView, and truncated summary, plus timeout behavior and jobId. It also guides next steps (create_strategy). Missing some details like exact error format but sufficient for a single-param tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% (single prompt parameter fully described with min/max and example). The description adds semantics by explaining what the prompt should contain ('what to research and from which angle') and gives an example, plus guidance to resolve tickers with search_symbols. This adds value beyond the schema's basic min/max.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Generate a saved AI research report' with specific elements (thesis, risks, catalysts, scored overall view) and for 'a company or asset from a natural-language prompt'. It distinguishes from siblings by mentioning 'saved' (vs get_research/list_research) and as a first step before building a strategy, referencing create_strategy.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit when to use: 'when the user wants fundamental or thematic research, or as the first step before building a research-grounded strategy' and mentions resolving tickers with search_symbols first. Also provides alternative: pass analysisId to create_strategy, and mentions get_job_status for timeout. Clear exclusions and alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_researchA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It usefully warns that narrative sections are truncated and that full layouts and raw data are never dumped, which are key behavioral traits. It doesn't explicitly state read-only status, but as a fetch operation this is largely implied; the added truncation warning adds significant value.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, front-loaded with the primary action, then usage context, then parameter and return information. Every sentence earns its place, and there is no waste or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (one parameter, no nested objects, no output schema). The description adequately explains return structure (title, symbol, score, view, truncated narrative sections) and key limitations, providing all essential context for an agent to use it correctly. No output schema means the description appropriately covers return values.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already documents analysisId as a UUID with the same origin hint (from create_research or list_research), providing 100% coverage. The description restates this information without adding new semantics, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Fetch') and resource ('saved research report by analysisId'), clearly distinguishing it from siblings like list_research (which lists reports) and get_strategy (which gets strategies). It explicitly identifies the key parameter and the operation's scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool: 'to re-read research created earlier' and 'before building or editing strategies'. It also names the alternative tool (list_research) for finding IDs, providing clear context and an alternative without ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_strategyA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It discloses the return format ('compact JSON') and an important negative behavior ('the raw graph is never dumped'), which helps set expectations. It stops short of stating read-only semantics or error conditions, but for a fetch operation this is reasonable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is tightly written: one sentence for purpose, one for usage context, one for parameters/return format. It front-loads the core action and every clause adds value, with no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter fetch tool with no output schema, the description fully covers what is returned, where to get the ID, and how it connects to optimize_strategy. It is sufficient for an agent to correctly select and invoke the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and the schema already explains the parameter ('Strategy UUID. Find it with list_strategies... Not a strategy name.'). The description adds only 'find it with list_strategies', which overlaps with schema text. No new meaning is added beyond the schema, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Fetch a single strategy's structure' and enumerates the specific contents (name, description, native timeframe, per-node summary), using a specific verb and resource. It also distinguishes from siblings by naming optimize_strategy as the downstream consumer and list_strategies for finding IDs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly warns 'Use this before optimize_strategy' and explains the exact relationship between node IDs/config field names and the xNodeId/xField parameters. It also mentions 'or to understand what a strategy actually does before editing it', providing clear when-to-use context and implicitly distinguishing from list_strategies.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• edit_strategyA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully discloses behavior: the 300s wait, the verification verdict comparing backtests, return fields including revertToVersion, verdict meanings, and timeout/error handling with jobId. This goes far beyond what annotations would typically provide and gives the agent a comprehensive understanding of side effects and failure modes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single dense paragraph but every sentence earns its place: purpose, parameter guidance, usage tips, return value explanation, verdict definitions, and timeout behavior. It is front-loaded with the core action and progressively details, which is appropriate given the complexity of the tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  There is no output schema, so the description must explain return values, and it does thoroughly: a verification verdict with before/after trade counts and return percentages, plus revertToVersion. It also covers all verdict values and the timeout scenario with jobId. For a 2-parameter tool with no annotations, this is complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already covers both parameters fully (100% coverage), so the baseline is 3. The description adds extra value by clarifying where to get strategyId (from list_strategies), suggesting that diagnose_strategy's agentRequest can be pasted verbatim into instruction, and advising to use get_capabilities for valid indicator names. This enrichment pushes it above baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Modify an existing strategy with a natural-language instruction using Pyon's AI editor', which is a specific verb+resource pairing. It further lists exact modification targets (entry/exit logic, thresholds, position sizing, symbols) and explicitly ties to fixing issues from diagnose_strategy, clearly distinguishing it from sibling tools like create_strategy or run_backtest.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states when to use the tool ('Use this to change...') and references sibling tools for complementary actions ('Use get_capabilities to check indicator names', 'check with get_job_status'). It does not explicitly list when NOT to use it, but the context is clear enough to guide selection among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_strategiesA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the return fields (id, name, nodeCount, updatedAt), the limit of 100 strategies, and points to get_strategy for more detail. It does not mention behavior beyond the 100 limit (e.g., ordering), but this is a simple read/list operation and the disclosure is strong enough.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences: the first states the purpose, the second gives usage guidance, the third covers parameters and return values. All information is essential, front-loaded, and free of fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with no mandatory parameters, the description is complete. It explains what it returns, how many results, and when to use a different tool for more detail. No output schema exists, but the description fully compensates.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has zero parameters, and the description explicitly states 'Parameters: none - the arguments object may be empty or omitted entirely.' This goes beyond the schema by confirming the parameters can be omitted entirely, which is helpful for an agent deciding how to invoke the tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource: 'List the user's saved trading strategies on Pyon.' It clearly distinguishes from siblings by explicitly naming which tools to use it before (get_strategy, run_backtest, etc.), making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit when-to-use guidance: 'Use this to find an existing strategy's id before calling...' and 'to check what already exists before building something new with create_strategy.' This clearly tells the agent when to use this tool versus alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_symbolsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It does: it limits results to 'up to 20 matches', specifies the return structure ('compact JSON with symbol, name, exchange, and assetClass'), and notes the fixed-universe limitation. It does not mention pagination or error handling, but for a simple lookup tool this is a solid disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but well-structured: it opens with the core search capability, follows with when-to-use and an illustrative example, then parameter constraints, return format, and a key caveat. Every sentence earns its place; no redundant phrasing or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with a single parameter, no output schema, and limited complexity, the description is complete. It covers the input, output format, result limit, and a critical limitation (fixed universe) with a pointer to the authoritative list. The agent can invoke this tool confidently with the expected result described well enough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds value by providing concrete examples ('resolving Apple to AAPL', 'checking whether Pyon covers a given asset') and restating the parameter range ('1-100 characters'). This helps the agent understand the intended use beyond the schema's literal definition.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Search Pyon's market database for tradable instruments by ticker or company name fragment.' It uses a specific verb (search), names the resource (market database), and indicates the query type (ticker or name fragment). It also differentiates from sibling tools by positioning this as the first step for resolving symbols, for example 'resolving Apple to AAPL'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit when-to-use guidance: 'Use this first whenever you need an exact symbol to mention in a research prompt or strategy description.' It also provides a concrete alternative: 'call get_capabilities(section='tickers') for the definitive tradable list,' which clarifies when not to rely solely on this tool (fixed universe). This exceeds baseline with clear context and exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_capabilitiesA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses silent failure modes, specific indicator ranges (RSI 0-100, WILLR -100 to 0), the literal node-graph interpretation, fallback to a bundled copy, and session caching—far beyond what annotations would typically convey.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long (~200 words) but dense with essential operational details and organized purpose-first. The parameter listing is slightly redundant with the schema, but every sentence provides critical guidance, so the length is justified.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a one-parameter catalog tool with no output schema and no annotations, the description is exceptionally complete. It covers return contents, error prone scenarios, parameter semantics, source fallback, caching, and fixed timeframes, leaving little ambiguity about tool behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already documents the single `section` parameter with 100% coverage, so the baseline is 3. The description adds extra context by explaining what each section contains and clarifying that timeframes are always 1m/5m/15m/1h/1d, going slightly beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Return the catalog of everything Pyon's strategy engine understands' and enumerates all catalog sections (indicators, operators, triggers, actions, timeframes, tickers, etc.), clearly distinguishing it from sibling tools like create_strategy or search_symbols.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit usage guidance is provided: 'CALL THIS BEFORE writing a create_strategy description, an edit_strategy instruction, or choosing optimize_strategy sweep bounds.' It also explains the consequence of not doing so (silently producing a broken strategy), which is actionable and clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_researchA
  4.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the return format ('compact list of up to 25 reports with analysisId, title, symbol, and timestamps') and parameter expectations ('arguments object may be empty or omitted'). However, it does not explicitly state the operation has no side effects (read-only), though 'List' strongly implies it. A explicit note would push this to 5.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences plus a parameter note, front-loaded with the core action and resource. Every phrase provides actionable information (purpose, usage, return details) with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a parameterless tool with no output schema, the description sufficiently covers return values (fields and limit), use cases, and relationship to sibling tools. It addresses all critical aspects an agent needs to invoke and interpret results correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has zero properties, so baseline is 4. The description adds valuable invocation guidance: 'Parameters: none - the arguments object may be empty or omitted entirely.' This clarifies that the agent can omit the arguments object altogether, which is not evident from the schema alone. Explicitly handling the absence of parameters earns a 5.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly specifies the action ('List') and the resource ('the user's saved research reports'), distinguishing it from siblings like list_strategies (strategies) and get_research (specific report retrieval). The phrase 'saved research reports' unambiguously identifies the scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use: 'to find an analysisId for get_research' and 'to check whether relevant research already exists before generating a new report with create_research'. This names sibling tools and provides concrete decision context, fully satisfying the dimension.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_backtestA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the 180-second wait, timeout behavior, diagnostic message contents, zero-trade nuance, short-window warning, and automatic timeframe fetching—all non-obvious behavioral traits beyond the schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but logically structured: action, workflow placement, parameters, return values, diagnostics, timeout fallback. While dense, every sentence carries useful information; only a slight trimming could improve directness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  There is no output schema, so the description fully enumerates return metrics, diagnostic messages, edge cases like holding positions, and timeout fallback. It also accounts for all 5 parameters and their defaults, making it complete for safe invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and each parameter already has a rich description. The description adds modest extra context like 'not a strategy name', 'fetched automatically', and the 7-day minimum gap, reinforcing but not dramatically extending schema info. This justifies a 4 above the 3 baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific action—'Backtest a strategy over a historical window and wait up to 180s'—clearly naming the verb and resource. It is distinguished from siblings like optimize_strategy and diagnose_strategy by focusing on strategy performance measurement over historical data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool: 'after create_strategy or edit_strategy to measure real performance, and before optimize_strategy to establish a baseline.' It also names get_job_status as a fallback on timeout, providing clear workflow context and alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• diagnose_strategyA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description fully discloses behavior: it runs a fresh backtest, waits up to 300s, returns a diagnosis, up to 10 issues, and an optional agentRequest. It also explains timeout error handling with jobId, which is crucial for an async-like operation. This is thorough and honest.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact yet dense: first sentence states purpose and core behavior, second gives usage context, third details parameters, fourth explains return value, fifth explains timeout. Every sentence serves a distinct purpose, and it is organized logically.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite no output schema and no annotations, the description covers the key aspects: what it does, when to use, parameters, return structure, timeout behavior, and how to use the result with edit_strategy. It also references sibling tools for finding strategyId. This is complete for a diagnostic tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% giving baseline 3, but the description adds valuable context: it explains the question is optional and gives an example, and clarifies that omitting it yields a general health check. It also reiterates the required nature of strategyId, adding practical usage nuance beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'analyze a strategy, running a fresh sample backtest as evidence' and distinguishes it from siblings like run_backtest by focusing on causal diagnosis. It explicitly mentions producing an edit request for edit_strategy, making its role unique.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives explicit when-to-use guidance: 'Use this when run_backtest shows zero trades, poor returns, or confusing diagnostics...'. It also provides a fallback for timeout via get_job_status, and implies not to use it if you only need a backtest. This is clear and actionable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_job_statusA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does so thoroughly. It discloses job id persistence ('stay resolvable indefinitely'), error semantics ('not-found error means the id is wrong'), return behavior (status, compact result, or error message), and advises retry after a wait if still running—all beyond basic 'get status'.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single dense paragraph, but every sentence earns its place—no fluff. It is front-loaded with the core purpose and then systematically covers usage, parameters, and behavior. Slightly long for a single-tool description, but well structured and informative.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Contextually complete: one parameter, no output schema, and no annotations. The description covers what the tool does, when to use it, parameter semantics, return values, error cases, and retry behavior. There is nothing meaningful missing for an agent to select and invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Although schema coverage is 100%, the description enriches the parameter meaning significantly: 'copy it verbatim from the timeout error message', the indefinite resolvability, and the backtest-id fallback. This goes well beyond the schema's simple text, making the parameter's intent and usage crystal clear.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Check a Pyon async job directly.' It explicitly distinguishes it from siblings by positioning it as the 'escape hatch when a waiting tool timed out' and explains which tools it applies to, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit when-to-use guidance: 'use when a waiting tool timed out' and enumerates the tools whose jobs can be checked. It also clarifies the run_backtest backtest-id fallback, providing clear context for when this tool is appropriate versus alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_strategyA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden and does so thoroughly: explains 300s wait/timeout behavior, return fields, error handling with jobId, and constraints on indicator thresholds. It also advises to run backtest before optimization, disclosing the expected workflow.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but every sentence adds value: purpose, usage, parameters, prerequisites, return value, next steps, and timeout handling are all covered without redundancy. Front-loaded with the primary action and structured logically.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a moderately complex tool with async behavior and dependencies, the description covers all essential aspects: what it does, when to use, prerequisites, parameters, return value, timeout handling, and follow-up actions. No significant gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Even though schema coverage is 100%, the description adds crucial semantic value: it explains what to include in the description parameter (instrument, entry/exit rules, position size), provides an example, and clarifies analysisId source. This goes beyond the schema's basic definitions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it builds a brand new trading strategy from natural language, using a specific verb (build), resource (trading strategy), and method (AI builder). This distinguishes it from siblings like list_strategies or edit_strategy.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'Use this when the user wants a new strategy' and provides concrete prerequisites (call get_capabilities first) and post-actions (evaluate with run_backtest). It also mentions optional grounding via analysisId, giving clear context for when to use this versus alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• optimize_strategyA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and excels: it discloses the 600s wait, timeout error with jobId, grid orientation, the 'preferring cells that actually traded' tie-break, and the verbatim zero-trade warning. It also explains default timeframe behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense and front-loaded, stating the core purpose first, then prerequisites, parameter rules, return values, and error handling. Every sentence adds operational value with no filler or repetition of schema text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For an 11-parameter, long-running optimization tool with no output schema, this description is thorough: it covers requirements, edge cases (zero trades), result contents, timeout behavior, and related tool calls. It leaves little room for misinterpretation or missing steps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Although schema coverage is 100%, the description adds substantial meaning: it explains the axis constraint (must not be same node id AND field), linear endpoint-inclusive spacing with a concrete example, bounds strictness (xMax > xMin), steps range/default, and how to source node ids/fields from get_strategy. This goes well beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names the specific action ('Run a 2-D parameter sweep'), the target resource (a strategy's nodes), and the method (backtests a steps x steps grid). It clearly distinguishes this from siblings like run_backtest by framing it as a tuning tool to use after a strategy is sensible.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use ('Use this after a strategy already trades sensibly'), prereqs (call get_strategy first, get_capabilities for bounds), and even warns against nonsensical ranges. It names alternative/companion tools (run_backtest, get_strategy, get_capabilities, get_job_status) and gives concrete conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.