compare evals
compare_evalsCompare two finished eval runs arm-by-arm: enter a baseline run and an after run to get per-arm delta, 95% interval, and a significance flag instead of reading two reports side by side.
Instructions
Compare two finished eval runs arm-by-arm (before vs after a prompt, tool or index change) and get each arm's delta with a 95% interval and a significance flag, instead of eyeballing two reports. GET /v1/evals/compare (API-key scope: read). Returns: {a:{id, name, created_at, judge_model, eval_kind}, b:{…same}, rows:[{arm (candidate key), metric ("win_rate" for comparison runs; "observed_pass_rate" and, when both runs carry one, "corrected_pass_rate" for criterion runs), a:{rate, n, ci:[lo,hi]}, b:{rate, n, ci}, delta (b.rate − a.rate), delta_ci:[lo,hi] (Newcombe 95%), significant (interval excludes zero)}], unmatched:{a:[arm keys only in a], b:[arm keys only in b]}}. Cache-Control: no-store. Notes: 400 when a or b is missing. 404 when either run is not in this workspace. 412 {code:"precondition_failed"} when both runs are not DONE, when they are different eval kinds, or when they used different judge models (a delta between judges measures the judges, not your change — re-run one with the other's judge). Best used with identical sample_filters on both runs. Read-only, no spend.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| a | Yes | Eval run id of the BEFORE run (baseline of the comparison). | |
| b | Yes | Eval run id of the AFTER run. Delta is reported as b − a. |