eval_compare_runs

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With no annotations provided, the description carries the full burden, and it does well: it discloses that files are loaded from disk, cases are paired by stable ID and case digest, McNemar's p-value is included only when pairing evidence supports it, and a large p-value does not establish equivalence. This prevents meaningful misinterpretation of statistical results and reveals the pairing behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but well-structured: purpose, usage, args, and returns are clearly separated and front-loaded. The detailed Returns section earns its place because it documents the delta semantics and field shapes, though it is slightly more verbose than strictly necessary.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given no annotations, minimal input schema, and complex return semantics, the description is complete. It covers input paths, pairing strategy, statistical caveats, return fields, and even pairing statistics like paired_count/added_count/removed_count so the agent can interpret drift between runs.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must compensate, and it does. It defines baseline_json_path as the filesystem path to the baseline report JSON with an example, and new_json_path as the new/proposal report JSON to compare. This adds semantic meaning and role beyond the raw string type in the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Does the description clearly state what the tool does and how it differs from similar tools?

The opening sentence states a specific action and resource: 'Compare two multivon-eval report JSONs and return a structured diff.' This clearly distinguishes eval_compare_runs from the sibling eval_* tools, which perform their own single-report evaluations rather than pairwise comparison.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly says when to use the tool: after a prompt/retrieval/model change when you want to know whether the new run improved on the baseline, case-by-case rather than only on aggregate. It does not explicitly name alternatives or state when-not-to-use, but the use-case guidance is clear enough to route an agent correctly.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.