Measure whether an LLM judge agrees with human labels beyond chance.

Reports Cohen's κ (weighted, for ordinal scores) with a bootstrap CI, next to the two numbers that make it legible: the majority-class baseline, and the inter-human ceiling. Accuracy alone is the trap — on a set that's 78% pass, a judge that always says "pass" scores 78%.

Args: judge_path: JSONL/CSV of judge outputs. Needs an id and a score/label/choice. gold_path: JSONL/CSV of human labels, joined on the same id. weights: "auto" | "linear" | "quadratic" | "none". Auto uses weighted κ for ordinal scales with >2 levels, which is nearly always what you want — plain κ treats 9-vs-8 as exactly as wrong as 9-vs-1. human_ceiling: inter-human κ, if you measured it. Without this we cannot tell a bad judge from an underdefined task. Also read from a rater_agreement field in the gold file. verbose: include evidence, fixes and citations.

Compare two eval runs and attribute the score change to the system or the judge.

Answers "my scores went up 6% — is that real?". Diffs the judge fingerprint (model, prompt hash, rubric hash, scale, temperature) between runs; if it changed, the two runs are not on the same scale. Then, using anchor items whose outputs are byte-identical across runs — so the system provably did not change — measures how much the judge itself moved and subtracts it.

Without anchors this reports the fingerprint change and refuses to apportion the delta, because apportioning it would assume the answer. Freeze ~30 items with fixed outputs, re-judge them every run: that's your judge canary.

Args: run_a_path: earlier run (JSONL/CSV). run_b_path: later run. anchor_ids: comma-separated item_ids of a declared frozen control set. Optional — anchors are auto-detected from identical output text. verbose: include evidence, fixes and citations.

Test an LLM judge for a specific bias.

Probes: position — swap A/B order; content identical, so any flip is order bias. Needs variant=""/"swapped" records per item_id. verbosity — pad outputs with content-free filler; any lift is bias. Needs variant=""/"padded". self_preference — does the judge favour its own family? Measured as residual against a human panel, since raw score gaps are confounded by real quality. Needs output_family + gold_path. length_confound — does the judge pay for length among answers humans rated equally? Distinguishes bias from "long answers are better". Needs gold_path + output lengths. sycophancy — does a content-free hint ("this is our new model") move the score? Needs variant=""/"hinted". distribution — no variants needed, runs on any log you already have. Finds granularity collapse, ceiling effects, dead rubric levels, and what your delta means in items-that-actually-moved.

Args: path: judge outputs, including probe variants where the probe needs them. probe: one of the above. gold_path: human labels (self_preference and length_confound require these). judge_family: e.g. "claude" — inferred from the judge model string if logged. scale_min/scale_max: declared rubric bounds for distribution. Pass these: inferring the scale from the judge's own output would define the "never uses the bottom half" pathology out of existence. claimed_delta: for distribution — a headline delta to translate into "how many items actually moved a notch". verbose: include evidence, fixes and citations.

Run every check this data supports, and say what's missing for the rest.

The "I have an eval log, is my judge okay?" entry point. Runs the free probes unconditionally, adds calibration and the human-controlled probes if gold labels are supplied, and reports which probes need variants you haven't generated yet — with the command to generate them.

Args: path: judge outputs. gold_path: human labels. Without these, roughly half the checks are unavailable, and the ones that need a quality control will say so rather than reporting a confounded number. judge_family: e.g. "claude". scale_min/scale_max: declared rubric bounds. verbose: full evidence for every finding (long).

Generate the variant file a probe needs your judge to score.

Writes a JSONL of items to judge (swapped orders / padded outputs / hinted prompts), each tagged with variant and keeping its original item_id. Run your own judge over it, fill in score/choice, then pass the result to bias_probe.

This round trip is why the core of this server needs no API key and costs nothing to run.

Args: path: an existing judge run to build variants from. probe: "position" | "verbosity" | "sycophancy". The others need no variants — distribution and length_confound run on your existing log directly. out_path: defaults to ..jsonl hint: for sycophancy — new_model | production | team_favourite | expensive.

Explain what a judge-audit metric means and why it's the right question.

Args: metric: kappa | anchors | fingerprint | effective_levels | position_bias | self_preference | length_confound | sycophancy | power | ceiling