audit_judge
Run all available checks on judge outputs to assess quality and identify which probes require missing variants for complete evaluation.
Instructions
Run every check this data supports, and say what's missing for the rest.
The "I have an eval log, is my judge okay?" entry point. Runs the free probes unconditionally, adds calibration and the human-controlled probes if gold labels are supplied, and reports which probes need variants you haven't generated yet — with the command to generate them.
Args: path: judge outputs. gold_path: human labels. Without these, roughly half the checks are unavailable, and the ones that need a quality control will say so rather than reporting a confounded number. judge_family: e.g. "claude". scale_min/scale_max: declared rubric bounds. verbose: full evidence for every finding (long).
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| path | Yes | ||
| verbose | No | ||
| gold_path | No | ||
| scale_max | No | ||
| scale_min | No | ||
| judge_family | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |