iris-eval/mcp-server

Compare Runs

compare_runs

Read-onlyIdempotent

Compare two runs to show whether a change made the agent worse, better, or equivalent, pairing cases statistically and flagging regressions and improvements.

Instructions

Did this change make the agent worse? Compares two runs: worse, better, equivalent, or too little evidence to tell.

What it does. Pairs by case_key (McNemar exact), else Newcombe. A rule counts only where it ran; no better if the second run was judged on less. Refuses unlike runs unless force. Local, no model call. Full detail: iris://capabilities.

When not to use it. To score one output (evaluate_output). As a gate: it reports; blocking is your policy.

Returns. JSON: comparable, incomparable_because, forced, method, before, after, difference, paired, worse, better, improvement_withheld, call, smallest_detectable, equivalent_within, coverage, critical_rises, rules_tested, regressions, improvements, discordant, discordant_total, summary, dataset.

Errors. IRIS_INVALID_ARGUMENT (empty run id, no baseline, bad margin or dataset); IRIS_STORAGE_ERROR. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. log_trace — record into a run; get_traces — find a run's traces.

Input Schema

Name	Required	Description
`after`	Yes	the run id to compare against it
`force`	No	Compare even when the runs are not strictly comparable; the response still names what changed
`before`	No	the baseline run id (`run` on log_trace); omit to use the run pinned with PATCH /api/v1/runs/:id
`dataset`	No	Restrict both runs to the case keys of this dataset (its id or label); every count then covers only those cases
`equivalence_margin`	No	δ for the equivalence test, as a difference in pass rate (0.05 = five points). Absent: equivalence is not tested; how far apart two runs may be and still count as the same is yours to choose

Output Schema

Name	Required	Description	Default
`call`	Yes
`after`	Yes
`worse`	Yes
`before`	Yes
`better`	Yes
`forced`	Yes
`method`	Yes
`paired`	Yes
`dataset`	Yes
`summary`	Yes
`coverage`	Yes
`comparable`	Yes
`difference`	Yes
`discordant`	Yes
`regressions`	Yes
`improvements`	Yes
`rules_tested`	Yes
`critical_rises`	Yes
`discordant_total`	Yes
`equivalent_within`	Yes
`smallest_detectable`	Yes
`improvement_withheld`	Yes
`incomparable_because`	Yes

Schema Changelog

Changes observed during successful MCP inspections.

Addedv0.10.1Oct 26, 2026

TDQS

A4.4/5.0

Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive; the description still adds real behavior beyond them (McNemar vs Newcombe pairing, counting only where a rule ran, withholding 'better' on thinner coverage, refusing unlike runs without force, local with no model call). That is substantive, but the error codes and recovery guidance are procedural boilerplate the agent could get from failure responses, so it sits mid-range rather than exceptional.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the decision question and organized under labeled sections (What it does, When not to use, Returns, Errors, Siblings), which makes it scannable. The ~22-item return field list is long and largely redundant with the output schema, costing some concision.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a five-parameter comparison tool with an output schema, the description still covers statistical method, refusal conditions, error surfaces, and sibling routing. Nothing an agent needs in order to choose and invoke it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3; the description earns above that by explaining the consequence of force on comparability and that dataset scoping restricts every count to those case keys. It leaves equivalence_margin's semantics almost entirely to the schema's own description.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with the decision question ('Did this change make the agent worse?') and names the specific operation and its outcome space: worse, better, equivalent, or too little evidence. The agent can distinguish it from compare_traces and evaluate_runs without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Contains an explicit 'When not to use it' block naming evaluate_output for single-output scoring and warning against using it as a gate, plus the condition that unlike runs are refused unless force is set. Alternatives and exclusions are stated, not implied.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server