LLM Evaluation Harness with MCP Tool Integration

An automated RAG evaluation harness scoring faithfulness, answer relevancy, and context precision — with a Gemini LLM-as-a-Judge backend (or optional RAGAS/DeepEval), exposed as a Model Context Protocol (MCP) server so any assistant can call it as tools, and wired into GitHub Actions for per-commit eval gating.

Why this exists

RAG pipelines silently regress. This harness turns evaluation into a gate: every commit is scored on the three metrics that matter for retrieval quality, and the build fails if any drops below threshold. The same eval logic is exposed over MCP so an agent can run evals as a tool — the part few portfolios show.

Related MCP server: hallumark

Metrics

Two backends: a self-contained Gemini LLM-as-a-Judge (default), or RAGAS/DeepEval (set USE_RAGAS=true).

Quickstart

1. Install

git clone https://github.com/SaiArja/llm-eval-mcp.git
cd llm-eval-mcp
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

2. Configure

cp .env.example .env
# set GEMINI_API_KEY=...   (https://aistudio.google.com/app/apikey)

3. Run the eval gate

python -m scripts.run_eval

The bundled data/sample_eval.jsonl contains three rows — one is a deliberately off-topic answer, so the aggregate drops the faithfulness/relevancy scores and the gate fails (exit code 1). Remove that row, or raise answer quality, and the gate passes. That pass/fail flip is exactly what runs in CI.

Get raw JSON:

python -m scripts.run_eval --json

Running as an MCP server

python -m app.mcp_server.server

This starts an MCP server over stdio exposing three tools:

Register it with any MCP client (e.g. Claude Desktop) by pointing the client at python -m app.mcp_server.server. An assistant can then evaluate RAG outputs on demand as a tool call.

CI gating (GitHub Actions)

.github/workflows/eval.yml runs unit tests and the eval gate on every push and PR. Add your key as a repo secret named GEMINI_API_KEY (Settings → Secrets and variables → Actions). A metric below threshold fails the build — turning "did this change hurt retrieval quality?" into a red/green check.

Dataset format

JSONL, one object per line:

{"question": "...", "answer": "...", "contexts": ["...", "..."], "ground_truth": "..."}

Project layout

app/
  eval/config.py     thresholds + backend selection
  eval/judge.py      Gemini LLM-as-a-Judge client
  eval/metrics.py    metric scoring (judge + optional RAGAS)
  eval/dataset.py    JSONL loader
  eval/runner.py     load -> score -> aggregate -> gate
  mcp_server/server.py  MCP tools
scripts/run_eval.py  CLI gate (exits non-zero on failure)
data/sample_eval.jsonl
.github/workflows/eval.yml
tests/               smoke tests (no key needed)

Tests

pip install pytest
pytest -q

License

MIT