A
licenseB
qualityB
maintenanceAn MCP-style stdio server for evaluating AI agent outputs, enabling CI-friendly quality gates, regression comparisons, and canary promotion decisions.
3
MIT
judge-audit-mcp
by asif786ka
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@judge-audit-mcpaudit drift between my two latest eval runs"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
judge-audit-mcp
Your eval scores went up 6%. Nobody checked whether the judge changed.
An MCP server that audits the LLM judge grading your evals: whether it agrees with humans, whether it drifted between runs, and whether it's biased. Everyone runs LLM-as-judge. Platforms ship judges. None of them ship judge QA.
$ judge-audit drift run_v1.jsonl run_v2.jsonl --fail-on serious
🛑 Judge changed between runs — 1 field(s) differ
These two runs were scored by different instruments. Their means are not on
the same scale, so the difference between them is not an improvement or a
regression — it is not a quantity at all until the judge's own shift is
measured and removed.
🛑 85% of the +6.0% score change is the judge, not the system
Headline: 6.873 → 7.287 (+6.0%) over 150 common item(s).
On 40 anchor item(s) — items whose output text is byte-identical in both
runs, so the system provably did not change — the judge scored +0.350
(+5.1%) higher in run B.
That movement is the judge's, by construction. Removing it leaves +0.063
for your system.
The judge accounts for 85% of the headline.
The improvement is the judge.
delta (judge shift on anchors): +0.350 95% CI [+0.200, +0.500] n=40
✗ failing: 2 finding(s) at or above seriousThat's the whole product. Someone bumped the judge model between two eval runs. Nobody wrote it down, because the judge isn't the thing under test — its version doesn't belong in the changelog of the system being evaluated. The 6% went in the launch doc. The system moved 0.9%.
This failure is silent (no error, the number is plausible), universal (everyone bumps judge models — the alternative is pinning a deprecated one forever), and unfalsifiable after the fact unless you happened to log the right things.
The idea: a score is comparable only within a fingerprint
A judge score is not a measurement of the thing being judged. It's a measurement
of the thing being judged as read by a particular judge under a particular
rubric — and the moment any part of that changes, the scale moves underneath
you and the numbers stop being commensurable. Scope is the fingerprint;
JudgeFingerprint.comparable_to(other) is the gate.
JudgeFingerprint(
model = "claude-sonnet-5-20260401", # "we upgraded" is a scale change
prompt_hash = "35c8032d", # PR: "clarify judge prompt"
rubric_hash = "ad8d493e", # someone tightened one criterion
scale = "1-10",
temperature = 0.0, # nonzero = not even repeatable
provider = "anthropic",
)There's no "minor" field in there on purpose. Every one of them is something people change without thinking of it as changing the measuring instrument. That's precisely why it goes unnoticed.
"Did my scores improve?" has no answer. It has an answer under a fixed fingerprint.
Related MCP server: groundcheck
Anchors: why the decomposition is an identity, not a model
You can't attribute a score change by comparing two means. mean(B) − mean(A)
confounds three things: the system changed, the eval set changed, and the judge
changed.
Joining on item_id kills the second. The third needs an anchor set: items
whose output is byte-identical across both runs. For those, the system provably
did not change, so any movement in their scores is the judge — wholly, and by
construction. That's not an assumption or a regression model. It's a difference
between two measurements of the same object.
judge_shift = mean(score_B − score_A) over anchors [system held fixed]
system_shift = total_shift − judge_shiftWithout anchors, this tool tells you the fingerprint changed and then refuses to apportion the delta, because apportioning it would assume the thing in question. That refusal is the honest answer, and the fix costs about a dollar:
Freeze ~30 items and their outputs. Re-judge them every run. That's your judge canary. Almost nobody has one, which is why almost nobody catches judge drift.
Accuracy is the trap
$ judge-audit calibrate judge.jsonl human_labels.jsonl
⚠️ Judge–human agreement: Cohen's κ = 0.381 (fair)
Raw agreement 83.0%, but 72.5% of that is expected by chance given the base
rates. A judge that always said 'pass' would score 78.5% on this set.
Cohen's κ: 0.381 95% CI [+0.219, +0.535] n=200
⚠️ Judge beats a constant-answer baseline by only +4.5%
83.0% accuracy sounds respectable until you notice that always answering
'pass' scores 78.5% on this set. The judge is contributing +4.5% over a rule
that reads none of the input.
ℹ️ Judge is 0.349 below the human ceiling
Humans agree with each other at κ=0.730; the judge reaches κ=0.381. There is
real headroom here — this gap belongs to the judge, not to the task.83% accurate reads fine in a slide. κ = 0.38 says it's barely doing better than a rule that reads none of the input. On any realistic eval set — where most items pass — accuracy flatters a broken judge, and it does so exactly when the benchmark is most representative.
The third finding is the one most tools skip. If your annotators only agree with
each other at κ=0.55, a judge at κ=0.50 is at the noise floor of the task and
"improve the judge" is the wrong project — the rubric is underdefined and no
judge, human or otherwise, can do better. Reporting κ without the ceiling sends
teams on month-long goose chases to fix a judge that's already as good as a
person. Pass --human-ceiling, or log rater_agreement in your label file, and
this tool will tell you which problem you actually have.
Six bias probes
Each asks: does the verdict move when something that shouldn't matter changes? The manipulation is orthogonal to quality by construction, so any response to it is bias by definition — there's no "well, maybe it was right" argument to have.
probe | manipulation | needs |
| swap A/B order | swapped variants |
| pad with content-free filler | padded variants |
| own-family vs other, vs a human panel |
|
| length at equal human quality | gold + lengths |
| a leading hint in the prompt | hinted variants |
| (none — reads your existing log) | nothing |
bias_probe(path, probe='distribution') costs zero extra judge calls and runs on
a log you already have. It's also the one that invalidates the most results:
🛑 Judge's effective resolution is 2.5 level(s), not 10
The judge used 3 distinct values: 7×62, 8×189, 9×49. Entropy 1.32 bits.
Everyone reading a score 'out of 10' believes they are looking at ten rungs
of measurement. They are looking at 2.5.
⚠️ 7 rubric level(s) never used: 1, 2, 3, 4, 5, 6, 10
Whatever your rubric says those levels mean has never influenced a score.
ℹ️ Your +0.430 delta is 129 item(s) hopping one notch
This judge emits whole numbers only. A +0.430 change is not 300 items each
improving slightly — it is ~129 items crossing a single threshold while the
other 171 did not move at all.The probe that proves the others aren't rubber stamps
Anyone can write a bias detector that always finds bias. The interesting question is whether it can be made to shut up.
length_confound exists to survive the obvious objection to verbosity: "of
course the judge scores long answers higher — long answers ARE better. Your bias
is the judge being right." On observational data, a correlation cannot refute
that. So control for quality with the human score:
judge_score ~ b0 + b1·human_score + b2·lengthb2 is what the judge pays for length among answers humans rated equally. If
length were merely a proxy for quality, b2 ≈ 0 — once you know how good an answer
is, its length tells you nothing more, and there's nothing left to pay for.
Two fixtures ship with this repo. Both have a near-identical naive correlation — the number people quote as proof of bias. The controlled slope separates them:
naive | controlled | verdict | |
| +0.815 | +1.454, CI [+0.70, +2.21] | 🛑 pays for length |
| +0.775 | +0.389, CI [−0.30, +1.11] | ✅ exonerated |
Same story from the naive number. Opposite truths. That's the whole argument for the probe — and every scenario in this repo ships with a control fixture like this, generated by the same mechanism with the bias removed. The tests assert both that the tool fires and that it stays quiet.
Findings carry their own epistemics
The bias literature reports effects that are real but modest — position flips in the 10-30% range, self-preference well under a point on a 10-point scale. Effects that size are trivially manufactured by sampling noise at n=20 plus a hopeful analyst. A tool that reports "κ = 0.61, substantial!" from 14 items is worse than no tool: it launders noise into a number with a Greek letter on it and someone puts it in a slide.
So every finding carries n, a bootstrap CI, and a tier that downgrades
itself:
certain— mechanical. "The judge model string differs between these runs" is a string comparison, not an inference, and needs no sample at all. The most damaging finding this tool makes is also the only one that needs no statistics.likely— the interval excludes the null and n clears the power floor.heuristic— underpowered or the CI straddles null. Never gates a build, whatever its severity.
temper_confidence() applies that rule automatically as evidence accumulates —
the claim isn't hand-edited, the model does it. A tool that red-builds on an
underpowered estimate gets switched off within a week — and then it audits
nothing at all.
Install
uvx judge-audit-mcp # MCP server (stdio)
pip install judge-audit-mcp # library + CLIOne dependency (mcp). The statistics are pure stdlib — Cohen's κ, weighted κ,
bootstrap intervals, exact binomial tests, OLS. scipy+sklearn would be ~80MB of
wheels to compute a 2×2 contingency table, and an audit tool that's annoying to
install doesn't get installed.
That trade is only defensible if the reimplementations are demonstrably right, so the smoke test cross-checks every estimator against scipy/scikit-learn when they're present and skips those assertions when they aren't:
pip install judge-audit-mcp[verify]
python smoke_test.py # 171 passing (160 without scipy — cross-checks skip)The dependency is optional for using this server and mandatory for doubting it, which is the correct way round.
Claude Desktop / Claude Code
{
"mcpServers": {
"judge-audit": {
"command": "uvx",
"args": ["judge-audit-mcp"]
}
}
}Tools
tool | question |
| Does the judge agree with humans beyond chance? (κ, baseline, ceiling) |
| Did my system improve, or did the judge change underneath me? |
| Is the judge biased? (six probes) |
| Generate the variants a probe needs me to judge |
| Run everything this data supports; say what's missing for the rest |
| What is κ / an anchor / a fingerprint, and why is it the right question? |
CI gate
- run: judge-audit drift baseline.jsonl current.jsonl --fail-on serious
- run: judge-audit calibrate judge.jsonl gold.jsonl --fail-on serious--format sarif renders findings as inline PR annotations on GitHub. Judge drift
becomes a red build instead of a wrong number in a launch doc.
Your data probably already works
The loader is aggressively tolerant, because everyone's eval log has a different
shape and a tool that demands a schema audits nobody's judge. JSONL, JSON arrays,
wrapped objects, CSV/TSV; nested fields flattened; key/q_id/example_id all
read as ids; outputs.score/rating/grade all read as scores; booleans coerced
to labels.
Guessing is always reported, never silent — reading latency_ms as your score
column and confidently reporting κ would be worse than failing.
One firm rule: item_id is required and never invented. Row order is not an
identifier, and pairing by it would fabricate the very join the conclusions rest
on.
The offline round trip
The probes need variants. This server doesn't call your judge — it writes the file of things to judge, you run your own judge over it, and you feed the results back:
judge-audit emit position my_eval.jsonl # → my_eval.position.jsonl
# ... score those with your judge ...
judge-audit probe position my_eval.position.jsonlThat round trip is why the core needs no API key, costs nothing, and produces the same numbers every time. An audit tool whose own results move between runs is in no position to lecture anyone about reproducibility.
A live adapter exists behind JUDGE_AUDIT_LIVE=1 + an API key for demos. It's a
convenience. The offline path is the product.
The fixtures are generated, not written
python fixtures/generate.pyEvery fixture comes out of a simulated process with a stated mechanism, not out of someone tuning a JSON file until the test went green. The generator says what it simulates and why the resulting number is the number — e.g. the drift fixture's judge upgrades ~30% of items by one point, and the +6.0% headline falls out of that. Change the mechanism and the headline moves; the tool's attribution tracks it either way, which is the property under test.
Seeded, so the files reproduce byte for byte. If a fixture ever needs to change, the diff shows the mechanism changing — the only honest way to edit a test fixture.
What this doesn't do
It can't tell you your judge is good. It can tell you it disagrees with your humans, drifted, or responds to things it shouldn't. A judge that passes every probe here may still be wrong in ways nobody has thought to probe for.
It inherits your labels' biases.
length_confoundmeasures the judge's excess length bias over your annotators'. If your humans are length-biased too, it under-reports.self_preferenceandlength_confoundneed human labels. Without them they report the confounded number and say, in as many words, that it isn't evidence. That's a refusal, not a limitation to be worked around — judge scores alone cannot separate "favours its own" from "its own was better", and more data will never fix it.Nonzero-temperature judges aren't repeatable at all, and this tool can only flag that, not quantify it, unless you supply repeat runs.
References
Cohen (1960), A Coefficient of Agreement for Nominal Scales — κ
Landis & Koch (1977) — the interpretation bands (a convention, not a law)
Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — position & verbosity bias
Wang et al. (2023), Large Language Models are not Fair Evaluators — order effects
Panickssery et al. (2024), LLM Evaluators Recognize and Favor Their Own Generations — self-preference
Dubois et al. (2024), Length-Controlled AlpacaEval — length control
Sharma et al. (2023), Towards Understanding Sycophancy in Language Models
License
MIT
Available Tools
6 toolsaudit_judgeA
Run every check this data supports, and say what's missing for the rest.
The "I have an eval log, is my judge okay?" entry point. Runs the free probes unconditionally, adds calibration and the human-controlled probes if gold labels are supplied, and reports which probes need variants you haven't generated yet — with the command to generate them.
Args: path: judge outputs. gold_path: human labels. Without these, roughly half the checks are unavailable, and the ones that need a quality control will say so rather than reporting a confounded number. judge_family: e.g. "claude". scale_min/scale_max: declared rubric bounds. verbose: full evidence for every finding (long).
ParametersJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| path | Yes | ||
| verbose | No | ||
| gold_path | No | ||
| scale_max | No | ||
| scale_min | No | ||
| judge_family | No |
Output Schema
ParametersJSON Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
A4.1/5.0
Behavior3/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description discloses that free probes run unconditionally and calibration probes are added with gold labels, but it does not explicitly state whether the tool has side effects, requires specific permissions, or is idempotent. Since no annotations are provided, the description carries the full burden, and this lack of detail leaves some gaps.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with a summary sentence, context paragraph, and Args list. It is clear and to the point, though slightly verbose for the context.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (6 parameters, conditional behavior) and the presence of an output schema, the description covers the overall workflow, including what happens under different conditions and what the output contains (reports missing probes with generation commands). It is sufficiently complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters4/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Despite 0% schema description coverage, the description's Args section adds meaning to all six parameters, explaining the purpose of each (e.g., gold_path enables half the checks, verbose provides full evidence). This compensates for the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states that the tool runs every check supported by the data and reports what's missing, positioning it as the entry point for auditing a judge. It distinguishes itself from siblings like bias_probe and calibrate_judge by specifying its comprehensive check-running role.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description specifies when to use (when you have an eval log and need to check if the judge is okay) and provides conditional behavior (gold labels trigger additional checks). However, it does not explicitly state when not to use or compare with alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
bias_probeA
Test an LLM judge for a specific bias.
Probes: position — swap A/B order; content identical, so any flip is order bias. Needs variant=""/"swapped" records per item_id. verbosity — pad outputs with content-free filler; any lift is bias. Needs variant=""/"padded". self_preference — does the judge favour its own family? Measured as residual against a human panel, since raw score gaps are confounded by real quality. Needs output_family + gold_path. length_confound — does the judge pay for length among answers humans rated equally? Distinguishes bias from "long answers are better". Needs gold_path + output lengths. sycophancy — does a content-free hint ("this is our new model") move the score? Needs variant=""/"hinted". distribution — no variants needed, runs on any log you already have. Finds granularity collapse, ceiling effects, dead rubric levels, and what your delta means in items-that-actually-moved.
Args:
path: judge outputs, including probe variants where the probe needs them.
probe: one of the above.
gold_path: human labels (self_preference and length_confound require these).
judge_family: e.g. "claude" — inferred from the judge model string if logged.
scale_min/scale_max: declared rubric bounds for distribution. Pass these:
inferring the scale from the judge's own output would define the
"never uses the bottom half" pathology out of existence.
claimed_delta: for distribution — a headline delta to translate into
"how many items actually moved a notch".
verbose: include evidence, fixes and citations.
ParametersJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| path | Yes | ||
| probe | Yes | ||
| verbose | No | ||
| gold_path | No | ||
| scale_max | No | ||
| scale_min | No | ||
| judge_family | No | ||
| claimed_delta | No |
Output Schema
ParametersJSON Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
A4.4/5.0
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations are provided, so the description carries full burden. It discloses that the tool measures specific biases, explains what each probe detects, and mentions that inferring the scale from the judge's own output would define the pathology out of existence. It also notes the 'verbose' flag includes evidence, fixes, and citations. The description is transparent about requirements and caveats.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is relatively long but well-structured with a clear opening sentence and bullet-pointed probes. The front-loading of the main purpose and parameter details helps an agent quickly grasp the tool. Some redundancy might be trimmed, but overall it is efficient for the complexity of the tool.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has 8 parameters (2 required) and an output schema. The description covers all probes, required parameters, and optional ones with explanations. It does not explicitly describe the return value format, but the presence of an output schema likely covers that. The description is complete enough for an agent to select and invoke the tool effectively.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, and it does. It explains the 'probe' parameter by detailing all six possible values with their meanings and data needs. It clarifies 'gold_path' for self_preference and length_confound, 'scale_min/max' for distribution, and 'claimed_delta' for interpreting distribution results. This adds substantial meaning beyond the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Test an LLM judge for a specific bias.' It then names and explains six distinct probes, each with a specific use case. This distinguishes it from sibling tools like audit_judge or calibrate_judge, which have different focuses.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines4/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit context for each probe, including required data variants and conditions (e.g., needs variant=''/''swapped'' records for position bias). It also notes that the 'distribution' probe runs on any existing log. However, it does not explicitly contrast the tool with siblings or state when not to use it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
calibrate_judgeA
Measure whether an LLM judge agrees with human labels beyond chance.
Reports Cohen's κ (weighted, for ordinal scores) with a bootstrap CI, next to the two numbers that make it legible: the majority-class baseline, and the inter-human ceiling. Accuracy alone is the trap — on a set that's 78% pass, a judge that always says "pass" scores 78%.
Args:
judge_path: JSONL/CSV of judge outputs. Needs an id and a score/label/choice.
gold_path: JSONL/CSV of human labels, joined on the same id.
weights: "auto" | "linear" | "quadratic" | "none". Auto uses weighted κ for
ordinal scales with >2 levels, which is nearly always what you want —
plain κ treats 9-vs-8 as exactly as wrong as 9-vs-1.
human_ceiling: inter-human κ, if you measured it. Without this we cannot
tell a bad judge from an underdefined task. Also read from a
rater_agreement field in the gold file.
verbose: include evidence, fixes and citations.
ParametersJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| verbose | No | ||
| weights | No | auto | |
| gold_path | Yes | ||
| judge_path | Yes | ||
| human_ceiling | No |
Output Schema
ParametersJSON Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
A4.3/5.0
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
No annotations exist, so the description carries full burden. It transparently describes outputs (kappa, bootstrap CI, baseline, ceiling) and warns about the accuracy trap. However, it does not explicitly disclose read-only or safety properties, though the statistical nature implies no side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with a clear introduction and then structured parameter documentation. While slightly verbose, every sentence adds value, and the format is easy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness5/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that an output schema exists (not shown but noted), the description adequately covers the tool's purpose, parameters, and key behavioral notes. No gaps remain for a calibration tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, but the description includes a detailed Args section that explains each parameter's purpose, default behavior (e.g., auto weights), and optional nature. This adds meaning far beyond the input schema alone.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool measures agreement between an LLM judge and human labels beyond chance, using Cohen's weighted kappa. This specific verb+resource distinguishes it from siblings like audit_judge or bias_probe.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implicitly conveys usage (when you have judge outputs and human gold labels), but does not explicitly state when to use this tool versus alternatives like audit_judge or detect_judge_drift. No when-not-to-use guidance is provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
detect_judge_driftA
Compare two eval runs and attribute the score change to the system or the judge.
Answers "my scores went up 6% — is that real?". Diffs the judge fingerprint (model, prompt hash, rubric hash, scale, temperature) between runs; if it changed, the two runs are not on the same scale. Then, using anchor items whose outputs are byte-identical across runs — so the system provably did not change — measures how much the judge itself moved and subtracts it.
Without anchors this reports the fingerprint change and refuses to apportion the delta, because apportioning it would assume the answer. Freeze ~30 items with fixed outputs, re-judge them every run: that's your judge canary.
Args: run_a_path: earlier run (JSONL/CSV). run_b_path: later run. anchor_ids: comma-separated item_ids of a declared frozen control set. Optional — anchors are auto-detected from identical output text. verbose: include evidence, fixes and citations.
ParametersJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| verbose | No | ||
| anchor_ids | No | ||
| run_a_path | Yes | ||
| run_b_path | Yes |
Output Schema
ParametersJSON Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
A4.3/5.0
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description covers the algorithm's logic, fingerprint comparison, anchor usage, and conditional output. It lacks explicit safety or permission info but is thorough for a read-only analysis tool.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the purpose, uses efficient sentences, and avoids redundancy. It includes both a high-level answer and necessary caveats without unnecessary detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the tool's logic, parameter behavior, and output expectations. An output schema exists, so return format details are not required. It is complete for a technical tool, though some edge cases (e.g., file format errors) are omitted.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, but the description provides clear semantics for all 4 parameters: run paths with format, anchor_ids with optionality and auto-detection, verbose with content list. This fully compensates for schema gaps.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool compares two eval runs and attributes score change to system or judge, answering a specific question ('is that real?'). It provides a detailed mechanism (diffing judge fingerprint, using anchor items) that distinguishes it from sibling tools like audit_judge or bias_probe.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines3/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explains the anchor requirement and behavior without anchors, giving context on when anchors are needed. However, it does not explicitly contrast with sibling tools or state when not to use this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
emit_probe_setA
Generate the variant file a probe needs your judge to score.
Writes a JSONL of items to judge (swapped orders / padded outputs / hinted
prompts), each tagged with variant and keeping its original item_id. Run your
own judge over it, fill in score/choice, then pass the result to bias_probe.
This round trip is why the core of this server needs no API key and costs nothing to run.
Args: path: an existing judge run to build variants from. probe: "position" | "verbosity" | "sycophancy". The others need no variants — distribution and length_confound run on your existing log directly. out_path: defaults to ..jsonl hint: for sycophancy — new_model | production | team_favourite | expensive.
ParametersJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| hint | No | new_model | |
| path | Yes | ||
| probe | Yes | ||
| out_path | No |
Output Schema
ParametersJSON Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
A4.6/5.0
Behavior4/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description carries full burden. It discloses that the tool writes a JSONL file to a configurable output path (defaulting to <path>.<probe>.jsonl), describes the output format, and explains that the server requires no API key. However, it does not explicitly state whether existing output files are overwritten or if there are any side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness4/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with a clear first sentence defining purpose, followed by output details, workflow, cost note, and argument list. It is somewhat verbose but each sentence adds value; a slight trim could improve conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the presence of an output schema and the complexity of the tool, the description adequately covers output format and workflow. It explains the round-trip and cost benefit. It could briefly mention the input format expected for path, but overall it is sufficiently complete for effective use.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description must compensate, which it does thoroughly. It explains each parameter: path (existing judge run), probe (enumeration with context on which probes need variants), out_path (default derivation), and hint (examples for sycophancy). This adds significant meaning beyond the bare schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool generates variant files for probe scoring, specifically writing a JSONL of items to judge with variant tags and original item_id. It distinguishes itself from siblings by noting which probes need variants and which don't (distribution, length_confound).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines5/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly explains the workflow: run emit_probe_set, then run a judge, then pass results to bias_probe. It specifies when to use (for position, verbosity, sycophancy probes) and when not (for distribution, length_confound), and references sibling bias_probe for the next step.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
explain_metricA
Explain what a judge-audit metric means and why it's the right question.
Args: metric: kappa | anchors | fingerprint | effective_levels | position_bias | self_preference | length_confound | sycophancy | power | ceiling
ParametersJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| metric | Yes |
Output Schema
ParametersJSON Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
A3.7/5.0
Behavior2/5
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With no annotations, the description must disclose behavioral traits. It states 'Explain' but does not confirm it is read-only, nor mentions any side effects, authentication needs, or output format. The presence of an output schema mitigates the lack of return description, but behavioral safety is unaddressed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Conciseness5/5
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise, front-loading the purpose in the first sentence. The args list is clearly formatted, and every element serves a purpose. No redundant or irrelevant information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Completeness4/5
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter explanation tool with an output schema, the description adequately covers the core purpose and valid inputs. It does not explicitly state that the output is a textual explanation, but the output schema likely handles that. One could argue for completeness given the tool's simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Parameters5/5
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Although schema description coverage is 0%, the description enumerates all valid values for the 'metric' parameter (kappa, anchors, fingerprint, etc.) within the args section, providing crucial context that the schema (which only defines a string) lacks. This fully compensates for the missing schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Purpose5/5
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states it explains 'what a judge-audit metric means and why it's the right question', specifying the resource (judge-audit metric) and distinguishing it from sibling tools that perform actions like auditing, probing, or calibrating.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Usage Guidelines2/5
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance on when to use this tool versus alternatives. The description implies it's for understanding metrics but does not provide explicit when-to-use or when-not-to-use instructions, nor does it mention alternative tools for related tasks.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
6 tool updates
v0.1.1- First observed
audit_judge - First observed
bias_probe - First observed
calibrate_judge - First observed
detect_judge_drift - First observed
emit_probe_set - First observed
explain_metric
TDQS
A4.2/5.0
Scored across 6 tools
Disambiguation5/5
Each tool serves a clearly distinct purpose: comprehensive audit, specific bias testing, calibration against humans, drift detection, data generation, and metric explanation. No two tools overlap in functionality.
Naming Consistency5/5
All tool names follow a consistent verb_noun pattern in snake_case (e.g., audit_judge, bias_probe, explain_metric). The convention is uniform and predictable.
Tool Count5/5
6 tools cover the essential aspects of LLM judge auditing without redundancy or bloat. Each tool is justified and serves a specific role in the workflow.
Completeness4/5
The set covers comprehensive auditing, bias probes, calibration, drift detection, data generation, and explanation. Minor gap: no tool for running a judge on raw inputs, but that is likely external to this server's scope.
Maintenance
ActivityStale
ResponsivenessNo issues
Related MCP Connectors
MCP-native AI evaluation: rubric audits, eval suites, and proof reports for AI/LLM output.
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
MCP server for building and testing AI agents with multi-model experimentation and insights.
Cloudflare Workers MCP server: llm-output-quality-monitor
Related MCP Servers
- AlicenseAqualityBmaintenanceAn MCP server that lets any AI agent evaluate RAG outputs -- faithfulness scoring, hallucination detection, and retrieval quality metrics -- with zero API keys, using MCP sampling.6MIT
- AlicenseAqualityCmaintenanceAn MCP server that runs curated adversarial prompts against local Ollama models to test guardrails, scoring responses with heuristic verdicts for human review.5MIT
- AlicenseBqualityCmaintenanceAn MCP server implementing the Flourishing-Justice-Autonomy (FJA) alignment framework, enabling FJA evaluation and fine-tuning of LLMs through examples like cultural diet, medical autonomy, and hiring fairness.2MIT