bias_probe
Test an LLM judge for specific biases by running controlled probes on its outputs. Uncovers position, verbosity, self-preference, length confound, sycophancy, and distribution issues.
Instructions
Test an LLM judge for a specific bias.
Probes: position — swap A/B order; content identical, so any flip is order bias. Needs variant=""/"swapped" records per item_id. verbosity — pad outputs with content-free filler; any lift is bias. Needs variant=""/"padded". self_preference — does the judge favour its own family? Measured as residual against a human panel, since raw score gaps are confounded by real quality. Needs output_family + gold_path. length_confound — does the judge pay for length among answers humans rated equally? Distinguishes bias from "long answers are better". Needs gold_path + output lengths. sycophancy — does a content-free hint ("this is our new model") move the score? Needs variant=""/"hinted". distribution — no variants needed, runs on any log you already have. Finds granularity collapse, ceiling effects, dead rubric levels, and what your delta means in items-that-actually-moved.
Args:
path: judge outputs, including probe variants where the probe needs them.
probe: one of the above.
gold_path: human labels (self_preference and length_confound require these).
judge_family: e.g. "claude" — inferred from the judge model string if logged.
scale_min/scale_max: declared rubric bounds for distribution. Pass these:
inferring the scale from the judge's own output would define the
"never uses the bottom half" pathology out of existence.
claimed_delta: for distribution — a headline delta to translate into
"how many items actually moved a notch".
verbose: include evidence, fixes and citations.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| path | Yes | ||
| probe | Yes | ||
| verbose | No | ||
| gold_path | No | ||
| scale_max | No | ||
| scale_min | No | ||
| judge_family | No | ||
| claimed_delta | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |