get eval gate
get_eval_gateConvert an eval run into a CI gate: returns 200 when all quality thresholds pass, 412 otherwise, to block unsafe model or prompt changes.
Instructions
Turn a finished eval run into a CI deploy decision with one call — 200 when every requested threshold passes, 412 otherwise — so a pipeline can curl -f it and block a bad model/prompt change. GET /v1/evals/{id}/gate (API-key scope: read). Returns: {pass: boolean, status: run status, checks:[{check: "win_rate"|"pass_rate"|"assertion_pass_rate"|"noninferiority", model, required, actual (CI lower bound or exact rate, null when unavailable), pass, note?}], reason? (set when the gate could not evaluate: run not DONE, run ERROR, or no thresholds given)}. HTTP 200 only when pass is true; 412 whenever anything failed. Notes: 412 (not 4xx-error shape — the verdict body itself) when: the run is not DONE ("Run not complete yet — poll until status is DONE." — fail closed), the run is ERROR, no threshold param was passed, or any check fails. Query values must parse as numbers in 0..1; anything else is treated as absent. 404 {error:{…}} when the run is not in this workspace. Keys in this response are NOT re-cased (they are already snake/single-word). Read-only, no spend. Pair it with a read-only scoped key for CI.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| id | Yes | The eval run id. | |
| model | No | Restrict the checks to one arm/model key (candidate key as listed in candidate_models). | |
| min_win_rate | No | 0..1. Comparison runs: every candidate arm's win-rate 95% CI LOWER bound must be ≥ this (never the point estimate). Each check names its basis: "corrected" when the run carries ≥30 human pair labels and a usable calibration (the number to gate on — the PRINTED win rate is compressed toward 50% by judge error), else "printed" with a note saying how to attach labels (label_eval_pair). When the corrected point clears the bar but the calibration floor cannot, the note prices the missing labels instead of asking for more samples. Fails closed when the judge returned no verdict on too many pairs (unreportable). | |
| min_pass_rate | No | 0..1. Criterion runs: every model's calibration-corrected pass-rate CI lower bound must be ≥ this; falls back to the observed CI when the judge is unvalidated (the check's note says so). | |
| win_rate_ties | No | Tie lever for the win-rate check. "half" (default): tie = half a win, parity 50%, comparable to the printed rate. "decided": ties dropped on both sides — wins/(wins+losses), the share among pairs someone decided; compresses less and needs fewer labels, answers a narrower question. Anything else is a 400. | |
| noninferiority_margin | No | 0..1. THE CERTIFIED SWITCH TEST: on a criterion run whose baseline is "__stored__" (the incumbent's logged answers) scored by a calibrated judge, each candidate's pass-rate CI floor must reach the incumbent's rate minus this margin (0.05 = provably within 5 points at worst). Requires the stored-baseline arm AND a calibrated judge (corrected rates) — no observed-rate fallback; fails otherwise with an explanatory note. | |
| min_assertion_pass_rate | No | 0..1. Every model's exact all-assertions pass rate must be ≥ this (deterministic count). Fails if the run has no assertions configured. |