detect_judge_drift
Compare two evaluation runs to determine if score changes are due to judge drift or system changes, using anchor items that are byte-identical across runs to isolate judge movement.
Instructions
Compare two eval runs and attribute the score change to the system or the judge.
Answers "my scores went up 6% — is that real?". Diffs the judge fingerprint (model, prompt hash, rubric hash, scale, temperature) between runs; if it changed, the two runs are not on the same scale. Then, using anchor items whose outputs are byte-identical across runs — so the system provably did not change — measures how much the judge itself moved and subtracts it.
Without anchors this reports the fingerprint change and refuses to apportion the delta, because apportioning it would assume the answer. Freeze ~30 items with fixed outputs, re-judge them every run: that's your judge canary.
Args: run_a_path: earlier run (JSONL/CSV). run_b_path: later run. anchor_ids: comma-separated item_ids of a declared frozen control set. Optional — anchors are auto-detected from identical output text. verbose: include evidence, fixes and citations.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| verbose | No | ||
| anchor_ids | No | ||
| run_a_path | Yes | ||
| run_b_path | Yes |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |