judge-mcp

English: README.en.md

LLM-as-Judge text quality evaluation as an MCP server.

Scores original/revised text pairs with a multi-axis weighted rubric, tests the judge model against a human-labeled golden set ("testing, not training"), and cross-validates agreement between two judges. Everything is exposed as MCP tools, so you can run evaluations conversationally from any MCP client — Claude Desktop, Claude Code, and more.

Why I built this

Proofreading, rewriting, and summarization have no single right answer, so more teams are turning to LLM judges. But an unvalidated judge is just another opinion. This server generalizes a methodology validated in a real production log-analysis project into a tool.

1. Multi-axis weighted rubric — meaning preservation, naturalness, improvement, and overcorrection restraint. Instead of an opaque single score, it scores per axis.

2. Golden set testing — tests judge candidates against human labels (good fix / missed / overcorrected). The golden set is never used for training, so it stays a clean exam; whenever the rubric changes, it's reused as a regression test.

3. Dual-judge cross-validation — scores the same items with two judges, measures agreement, and empirically shows whether conclusions are robust to judge choice.

4. Deterministic prefilter — no-change pairs are scored by code, no LLM needed. Cheap, and always the same answer.

Related MCP server: mcp-eval-harness

Tools

A synthetic Korean proofreading golden set of 24 items (3 buckets) is built in, and you can supply your own golden set (JSONL: id, original, revised, label) via golden_path.

Install & configure

Works with any OpenAI-compatible backend — OpenAI, local vLLM, Ollama, LM Studio.

pip install -e .

export JUDGE_API_BASE="https://api.openai.com/v1"   # vLLM이면 http://localhost:8000/v1
export JUDGE_API_KEY="sk-..."                        # 로컬 서버는 아무 값
export JUDGE_MODEL="gpt-4o-mini"                     # 또는 Qwen/Qwen2.5-7B-Instruct-AWQ

A Claude Desktop config example is in examples/. After configuring, you can ask things like:

• "Score this edit: original '오늘 회의 몇시에 시작하나요', revised '몇 시에 시작하나요?'"

• "Run the golden set with gpt-4o-mini and show me the report card"

• "Compare whether gpt-4o-mini and local Qwen agree as judges"

Measured results

Results from testing the built-in 24-item golden set with Claude Sonnet (claude-cli backend):

Reproduce it yourself: python scripts/run_golden.py --backend claude-cli (for OpenAI-compatible APIs, use --backend api).

Case study: the golden set caught a design flaw

The first run scored 62.5% — the missed bucket was 0/8. The cause wasn't the judge but the verdict design. A missed edit preserves meaning and has no overcorrection, so the two highest-weighted axes (0.40+0.35) score full marks, and even with an improvement axis of 1, the composite lands at 8–9. Catching misses via the composite score was mathematically impossible.

Fix: changed the missed-bucket verdict from the composite score to improvement axis ≤ 4. Retest: 62.5% → 83.3%, missed 0/8 → 7/8. This is exactly why the golden-set regression loop exists — it surfaced a quietly hidden evaluation design flaw with real measurements, fixed it, and locked it in with a regression test. (Remaining gap: the overcorrected bucket shows run-to-run variance in LLM judging — a candidate for few-shot anchor reinforcement, improvable through the same regression loop.)

Design notes

• Judge validation is the core. Judge models start with polar-opposite biases by default, so picking one without testing is a guess. Accuracy is judged by direction — well-fixed edits score high (≥7), missed and overcorrected edits score low (≤5). The gray zone (5–7) is deliberately counted as wrong.

• Reproducibility first. temperature 0, forced JSON output with parsing that tolerates code fences and chatter, score-range clamping, and cases that need no model (no-change) go through the code path.

• Actionable errors. Connection and schema failures return messages that say "what to fix" rather than stack traces.

Tests

pip install -e ".[dev]"
pytest   # 21건, 네트워크 불필요 (모의 LLM)

License

MIT