calibrate_judge
Measure LLM judge agreement with human labels using Cohen's kappa with bootstrap CI, plus majority baseline and inter-human ceiling to contextualize accuracy and detect chance-level performance.
Instructions
Measure whether an LLM judge agrees with human labels beyond chance.
Reports Cohen's κ (weighted, for ordinal scores) with a bootstrap CI, next to the two numbers that make it legible: the majority-class baseline, and the inter-human ceiling. Accuracy alone is the trap — on a set that's 78% pass, a judge that always says "pass" scores 78%.
Args:
judge_path: JSONL/CSV of judge outputs. Needs an id and a score/label/choice.
gold_path: JSONL/CSV of human labels, joined on the same id.
weights: "auto" | "linear" | "quadratic" | "none". Auto uses weighted κ for
ordinal scales with >2 levels, which is nearly always what you want —
plain κ treats 9-vs-8 as exactly as wrong as 9-vs-1.
human_ceiling: inter-human κ, if you measured it. Without this we cannot
tell a bad judge from an underdefined task. Also read from a
rater_agreement field in the gold file.
verbose: include evidence, fixes and citations.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| verbose | No | ||
| weights | No | auto | |
| gold_path | Yes | ||
| judge_path | Yes | ||
| human_ceiling | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |