iris-eval/mcp-server

Compare Traces

compare_traces

Read-onlyIdempotent

Measure agent reliability across repeated evaluations by grouping them into cases or sessions and reporting pass rates, flaky cases, and repeat-aware intervals.

Instructions

How reliably does the agent answer the same question? Per-case pass rates, flaky cases, and an interval that respects repeats.

What it does. Groups evaluations by case_key (or session) and reports each case's pass rate with a Wilson interval, the flaky cases least reliable first, and an overall rate over cases, with an exact interval, rather than over pooled attempts. Deterministic, local, no model call. Full detail: iris://capabilities.

When not to use it. To compare two runs (compare_runs). To score an output (evaluate_output).

Returns. JSON: case_key, group_by, question, cases, attempts, overall, pooled, flaky_cases, by_case, summary.

Errors. IRIS_STORAGE_ERROR. No matching evaluations is cases 0, not an error. Failures return {"error":{code,message,recovery}} with isError; follow recovery.

Siblings. compare_runs — compare two runs against each other; log_trace — record an execution with a case_key; get_traces — read the traces themselves.

Input Schema

Name	Required	Description
`run`	No	narrow to one run; omit to read every evaluation that carries a case key
`session`	No	narrow to one session (the turns logged with that session_id)
`case_key`	No	narrow to a single case — the fastest way to ask "is this one question flaky?"
`group_by`	No	case_key (default): repeats of one question; session: the turns of one conversation, so a session answered both ways reads as flaky and by_case rows carry session ids
`question`	No	read the rate for ONE question: only evaluations that judged it count, and an attempt passes when every rule answering it passed — not the composed verdict
`min_attempts`	No	ignore cases asked fewer than this many times (default 1). A case asked once cannot be shown to be flaky

Output Schema

Name	Required	Description	Default
`cases`	Yes
`pooled`	Yes
`by_case`	Yes
`overall`	Yes
`summary`	Yes
`attempts`	Yes
`case_key`	Yes
`group_by`	Yes
`question`	Yes
`flaky_cases`	Yes

Schema Changelog

Changes observed during successful MCP inspections.

Addedv0.10.1Oct 26, 2026

TDQS

A4.6/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/non-destructive/closed-world, but the description adds genuinely new behavior: deterministic and local with no model call, IRIS_STORAGE_ERROR as the failure mode, the non-obvious fact that zero matching evaluations returns cases=0 rather than an error, and the {error:{code,message,recovery}} envelope with isError. That is exactly the operational context annotations cannot express.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Well front-loaded with the core question, then labeled sections (What it does / When not to use it / Returns / Errors / Siblings) that make it skimmable. Slightly redundant: the Siblings line largely repeats the 'When not to use it' exclusions, and the Returns field list duplicates the output schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a six-parameter analytics tool with an output schema, the definition covers purpose, exclusions, failure shape, the empty-result edge case, and a pointer to iris://capabilities for full detail. An agent has everything needed to select and call it correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the enum parameters are documented in-schema, so the baseline is 3. The description restates the group_by dichotomy (case_key vs session) and 'over cases rather than pooled attempts,' which mirrors what the schema already says and adds no syntax or edge-case guidance for run, session, min_attempts, or question.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Opens with the concrete question the tool answers (per-case pass rates, flaky cases, interval respecting repeats) and then states the mechanism: grouping evaluations by case_key or session and reporting Wilson intervals per case plus an exact interval over cases. This is a specific verb+resource+method, clearly distinct from compare_runs (two runs against each other) and evaluate_output (score one output).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

An explicit 'When not to use it' section routes the agent to compare_runs for run-vs-run comparison and evaluate_output for scoring a single output, and the Siblings section reinforces the boundary with log_trace and get_traces. The condition that selects this tool over its alternatives is unambiguous.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server