Server Quality Checklist

Tool Scores

• aurora_simulateA
  3.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden, and it does disclose an important non-obvious behavior: it pauses rather than extrapolating when confidence intervals grow too wide, and it uses dynamics fitted and validated on the data. It does not clarify what a pause looks like in the output or whether the operation mutates anything, but the key behavioral guarantee is present.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no filler. The core mechanism and purpose are front-loaded, and the pause behavior is integrated naturally rather than tacked on.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given there is no output schema and no annotations, the description should be more self-sufficient. It leaves the required path parameter undefined, does not explain ci_pause_threshold, and does not describe what is returned when the tool pauses. It is enough to select the tool but not enough to confidently invoke it without additional assumptions.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must explain parameters. It gives meaning to n_steps and target_entity_id, but the required path parameter is never explained, and ci_pause_threshold is only indirectly inferable from the pause behavior rather than explicitly defined.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific operation—simulate the system forward n_steps—and scopes it to 'trajectory questions' on a completed run, optionally for a specific node. It is clear enough to distinguish from analyze/explain, but it does not explicitly contrast itself with aurora_forecast, so it stops short of full sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives a clear when-to-use context: trajectory questions on a completed run, with the optional target_entity_id instruction. It does not name alternative sibling tools or provide exclusion criteria, so it lacks the explicit when-not/alternative guidance needed for a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• aurora_load_bundleA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses verification behavior (SHA-256 and Ed25519) and the summary return values. However, it does not state what happens when verification fails, whether the operation is read-only, or any side effects, which are important for a file-loading tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences with no filler. It front-loads the core action and verification requirement, then gives usage context, then return values. Every sentence contributes essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has moderate complexity with 2 parameters, no output schema, and no annotations. The description covers the main purpose and return values, but omits failure handling and the behavior of the 'verify' parameter. An agent calling this tool would still need to infer the effect of verify=false, which is a notable completeness gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It implies 'path' points to the bundle file, but it does not explain the 'verify' boolean parameter's meaning or that it defaults to true and could potentially be set false. This is a significant gap because the description emphasizes verification without mapping to the parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Load' and resource '.aurora.json analysis bundle', and clearly states its purpose: verify integrity before trusting findings. It also lists the return values (run identity, confidence, fabricated_count, findings-by-severity counts), which distinguishes it from the sibling analysis tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly provides a use case: 'Use when someone shares an Aurora bundle and you need proof it is untampered.' This gives clear context for when to call the tool. It does not mention alternatives or exclusion scenarios, but the 'BEFORE trusting its findings' hint implies it should precede analysis tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• aurora_forecastA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries the burden. It discloses that the forecast is model-based, fitted and validated on actual data, that the method is disclosed, and that output is either point predictions with an honest horizon or a peak within horizon_hours. Side effects, failure modes, and result structure are not covered, but the core behavior is transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no filler. The core behavior is front-loaded and the usage hint wraps up efficiently; every clause contributes information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is sufficient for choosing the tool but not fully complete for invoking it: path's meaning is unresolved and the relationship between return_peak and horizon_hours (e.g., whether horizon_hours is required for peak mode) is underspecified. With no output schema or annotations, a bit more detail on expected inputs and return shape would make it complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds meaning for return_peak ('just the peak within horizon_hours') and implies horizon_hours controls the forecast window, but the required path parameter is left unexplained. This is a meaningful gap since the agent cannot tell what path refers to.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a concrete resource and operation: model-based forecast for the run's target column, returning point predictions or a peak. It clearly conveys what the tool does and is distinguishable by function, though it does not name or differentiate against sibling tools like aurora_explain or aurora_simulate.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit user intent: 'what will X be / when does it peak' questions, with a directive 'Use for... instead of extrapolating by eye.' This is clear context for when to invoke, but there are no when-not conditions or named alternative tools, so it stops short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• aurora_interveneA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description itself carries the behavioral burden and does so reasonably: it discloses the mechanism (one-variable perturbation, propagation up to max_depth hops) and the return shape (per-node deltas with confidence intervals). It does not explicitly state that data is unmodified or describe prerequisites, but the 'what-if' framing and computed-deltas wording largely convey the read-only analytical intent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two dense sentences with the operation front-loaded and the use case in the second sentence. Every clause adds information: mechanism, output type, usage context, and evidential basis, with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a sparse schema with no annotations and no output schema, the description covers purpose, usage, and output type well. But it omits the meaning of the required path parameter, the role of source_entity_id, and any prerequisite like having a discovered/loaded model available, so an agent may still be uncertain about how to assemble a valid call.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  At 0% schema description coverage, the description must compensate. It usefully explains max_depth as 'hops' and perturbation as 'Δ', and 'perturb one variable' loosely maps to source_entity_id. However, the required path parameter is never semantically clarified, so the required parameter set is only partially documented.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly identifies a specific operation: what-if causal intervention on a discovered model, perturbing one variable and propagating through validated relationships. It is distinct from forecast/simulate by emphasizing 'data's own causal graph' and 'not from priors', but it does not explicitly name or contrast a sibling, so it stops short of a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly gives a use-case template ('what happens to Y if X changes by Δ'), which tells an agent when to select this tool. However, it offers no exclusions or named alternatives among the sibling group, leaving routing to aurora_simulate or aurora_forecast partly to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• aurora_explainA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and does a good job: it reveals the tool returns full evidence, computed values, assumptions, parameters, and references, and clarifies it is not a summarized response. It does not explicitly state that the operation is read-only, but the 'evidence/receipt' framing makes an informational read behavior clear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded: the core action and target ('Full evidence for ONE finding by claim_id') appear first, followed by contents and a crisp usage rule. Every sentence earns its place with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The return content is described well enough at a high level, and the tool is not complex, but the absence of any explanation for the required path parameter is a real gap. Since there is no output schema and no annotations, the description should have clarified both inputs to be fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate for the bare parameter names, but it only clarifies claim_id ('ONE finding by claim_id'). The required path parameter is never explained, leaving the agent to guess whether it is a bundle path, workspace path, or something else.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('explain') with a precise resource ('ONE finding by claim_id') and clearly states what is delivered: computed values plus the method's registry spec. The phrase 'this is the receipt, not a summary' explicitly distinguishes it from a higher-level findings tool, so an agent can separate it from siblings like aurora_findings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives an explicit condition for use: 'Use whenever you are about to cite, verify, or defend a specific statistical claim.' This is clear context but it does not name alternative tools or state when not to use it beyond the implicit contrast with a summary.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• aurora_analyzeA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure, and it does so richly: read-only, local, on-device execution, cited findings with claim_id, contractually zero fabricated_count, changepoint calibration blocks, and verdict downgrade behavior. It does not discuss error conditions or limits, but it goes far beyond a minimal description. No contradiction with annotations because none exist.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is dense but well-structured: purpose and formats lead, followed by methods, return characteristics, and finally usage guidance. Each sentence carries substantive information; the only mild redundancy is the closing list ('analyze data, find anomalies, check what changed'), which is acceptable as a routing hint. It is longer than typical, but length is justified by the richness of the tool's behavior.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and no annotations, this description covers input types, return payload (cited findings, confidence, fabricated_count), behavioral guarantees (calibration, downgrade), and usage routing. The only notable omission is parameter semantics for depth and sections. For a statistically complex tool, this is strong coverage — an agent can invoke it correctly without opening the schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 25%, meaning depth, sections, and full_bundle are undocumented in the schema. The description clarifies full_bundle ('compact summary unless full_bundle=true') and partially clarifies path ('CSV path, existing run_dir, or .aurora.json'), but it does not explain the semantic meaning of depth (auto/quick/standard/full) or sections. The description compensates somewhat but not enough to fully bridge the parameter meaning gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Statistical analysis'), a clear resource ('dataset CSV, TSV, Parquet, XLSX'), and enumerates concrete capabilities (anomaly detection, changepoint detection, forecasting, causal discovery). It strongly distinguishes this tool from siblings by positioning it as the primary analysis entry point, name-dropping methods that no other sibling claims.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit when-to-use guidance: 'Use this FIRST whenever a user asks to analyze data, find anomalies, check what changed, or wants real statistics instead of estimates.' This directly tells the agent when to invoke this tool and implicitly routes around alternatives like aurora_forecast or aurora_simulate. This is exactly the kind of usage context an agent needs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• aurora_findingsA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations available, the description carries the full transparency burden. It clearly discloses what the returned findings contain and that they are 'verified,' so the agent knows this is an enumeration/read operation rather than a mutation. It does not mention ordering, default limit, or empty-result behavior, but the core behavior is well described.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact yet information-dense: two sentences covering purpose, output shape, filtering, sequencing, and usage guidance. It front-loads the core action and does not waste words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a listing tool with no output schema, the description provides strong context: what to do first, what results contain, how to filter, and how to follow up via aurora_explain. It lacks explicit mention of limit behavior and result ordering, but otherwise the agent has enough to invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, so the description must compensate. It defines path as 'an Aurora run or bundle,' method as 'statistical method,' and severity with its allowed values ('crit/warn/info'). It does not explain the limit parameter, but the name and schema constraints make it reasonably inferable.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with the specific verb 'List' and identifies the exact resource: 'verified findings from an Aurora run or bundle.' It enumerates the key output attributes (severity, method, threshold, citation, claim_id), which makes the tool's purpose unmistakable and distinguishes it from siblings like aurora_explain and aurora_analyze.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells the agent when to use this tool: 'Use after aurora_analyze to enumerate what was actually found.' It also gives a concrete behavioral directive—'quote findings from here instead of paraphrasing from memory'—and points to aurora_explain for evidence drill-down, effectively routing between alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.