label eval pair
label_eval_pairRecord a human preference verdict for a candidate-baseline sample to build calibration evidence, producing a corrected win rate with uncertainty interval.
Instructions
Record a HUMAN's verdict on one candidate-vs-baseline pair of a comparison run — the calibration evidence behind the corrected win rate. A pairwise judge's printed win rate is compressed toward 50/50 (a true 80/20 prints ~70/30 even for a judge at the human ceiling); from 30 labels the run reports a corrected rate with an interval that carries the calibration uncertainty. POST /v1/evals/{id}/pair_labels (API-key scope: evals:write). Returns: {sample_index, candidate, verdict, critique, pairwise} — pairwise is the run's refreshed calibration block (same shape as get_eval_pairwise), so one call shows what the label bought. One label per (sample, candidate); posting again overwrites. DELETE /v1/evals/{id}/pair_labels?sample_index=…&candidate=… removes one; GET lists them. Notes: Labelling is a human's job: only relay verdicts the user actually gave — never invent preferences to reach 30. 400 with the offender named on a bad sample_index/candidate/verdict; 404 when the run is not in this workspace. No spend.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| id | Yes | The eval run id (a comparison run; criterion runs are refused — they are graded pass/fail per exchange with trace labels). | |
| verdict | Yes | Which answer the human preferred: "candidate", "baseline", or "tie" (a tie is a real answer, not a skip). | |
| critique | No | Optional free-text WHY (≤2000 chars). | |
| candidate | Yes | The candidate arm key the verdict is about (as listed in candidate_models / get_eval_samples). | |
| sample_index | Yes | 0-based sample index within the run (the order get_eval_samples returns). |