Allows local Ollama models to serve as the LLM backend for the evaluation server, supporting text quality scoring, golden set testing, and model comparison.
Enables using OpenAI-compatible APIs as the LLM backend for text quality evaluation, including multi-axis scoring, golden set testing, and model comparison.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@judge-mcp Score this edit: 'I am happy' -> 'I'm thrilled'"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
English: README.en.md
LLM-as-Judge text quality evaluation as an MCP server.
Scores original/revised text pairs with a multi-axis weighted rubric, tests the judge model against a human-labeled golden set ("testing, not training"), and cross-validates agreement between two judges. Everything is exposed as MCP tools, so you can run evaluations conversationally from any MCP client — Claude Desktop, Claude Code, and more.
Proofreading, rewriting, and summarization have no single right answer, so more teams are turning to LLM judges. But an unvalidated judge is just another opinion. This server generalizes a methodology validated in a real production log-analysis project into a tool.
Multi-axis weighted rubric — meaning preservation, naturalness, improvement, and overcorrection restraint. Instead of an opaque single score, it scores per axis.
Golden set testing — tests judge candidates against human labels (good fix / missed / overcorrected). The golden set is never used for training, so it stays a clean exam; whenever the rubric changes, it's reused as a regression test.
Dual-judge cross-validation — scores the same items with two judges, measures agreement, and empirically shows whether conclusions are robust to judge choice.
Deterministic prefilter — no-change pairs are scored by code, no LLM needed. Cheap, and always the same answer.
Related MCP server: mcp-eval-harness
Tool | Function |
| Scores one original/revised pair → per-axis scores, weighted composite, good/fair/poor verdict, rationale |
| Tests the judge on the golden set → overall and per-bucket accuracy report |
| Dual-scoring with two models → ±1 agreement rate, low-quality verdict agreement, mean difference |
| Lists rubrics with axes and weights |
A synthetic Korean proofreading golden set of 24 items (3 buckets) is built in, and you can supply your own golden set (JSONL: id, original, revised, label) via golden_path.
Works with any OpenAI-compatible backend — OpenAI, local vLLM, Ollama, LM Studio.
pip install -e .
export JUDGE_API_BASE="https://api.openai.com/v1" # vLLM이면 http://localhost:8000/v1
export JUDGE_API_KEY="sk-..." # 로컬 서버는 아무 값
export JUDGE_MODEL="gpt-4o-mini" # 또는 Qwen/Qwen2.5-7B-Instruct-AWQA Claude Desktop config example is in examples/. After configuring, you can ask things like:
"Score this edit: original '오늘 회의 몇시에 시작하나요', revised '몇 시에 시작하나요?'"
"Run the golden set with gpt-4o-mini and show me the report card"
"Compare whether gpt-4o-mini and local Qwen agree as judges"
Results from testing the built-in 24-item golden set with Claude Sonnet (claude-cli backend):
Bucket | Accuracy |
good fix (good_fix) | 8/8 |
missed (missed) | 7/8 |
overcorrected (overcorrected) | 5/8 |
Overall | 83.3% |
Reproduce it yourself: python scripts/run_golden.py --backend claude-cli (for OpenAI-compatible APIs, use --backend api).
The first run scored 62.5% — the missed bucket was 0/8. The cause wasn't the judge but the verdict design. A missed edit preserves meaning and has no overcorrection, so the two highest-weighted axes (0.40+0.35) score full marks, and even with an improvement axis of 1, the composite lands at 8–9. Catching misses via the composite score was mathematically impossible.
Fix: changed the missed-bucket verdict from the composite score to improvement axis ≤ 4. Retest: 62.5% → 83.3%, missed 0/8 → 7/8. This is exactly why the golden-set regression loop exists — it surfaced a quietly hidden evaluation design flaw with real measurements, fixed it, and locked it in with a regression test. (Remaining gap: the overcorrected bucket shows run-to-run variance in LLM judging — a candidate for few-shot anchor reinforcement, improvable through the same regression loop.)
Judge validation is the core. Judge models start with polar-opposite biases by default, so picking one without testing is a guess. Accuracy is judged by direction — well-fixed edits score high (≥7), missed and overcorrected edits score low (≤5). The gray zone (5–7) is deliberately counted as wrong.
Reproducibility first. temperature 0, forced JSON output with parsing that tolerates code fences and chatter, score-range clamping, and cases that need no model (no-change) go through the code path.
Actionable errors. Connection and schema failures return messages that say "what to fix" rather than stack traces.
pip install -e ".[dev]"
pytest # 21건, 네트워크 불필요 (모의 LLM)MIT
This server cannot be deployed
MCP-native AI evaluation: rubric audits, eval suites, and proof reports for AI/LLM output.
Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.
Pay-per-call AI evaluation MCP server. Score LLM outputs against benchmark rubrics via Workers AI.
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
