# LLM Evaluation Harness with MCP Tool Integration

> An automated RAG evaluation harness scoring **faithfulness**, **answer
> relevancy**, and **context precision** — with a Gemini **LLM-as-a-Judge**
> backend (or optional RAGAS/DeepEval), exposed as a **Model Context Protocol
> (MCP) server** so any assistant can call it as tools, and wired into **GitHub
> Actions** for per-commit eval gating.

![architecture](docs/architecture.svg)

## Why this exists

RAG pipelines silently regress. This harness turns evaluation into a gate: every
commit is scored on the three metrics that matter for retrieval quality, and the
build fails if any drops below threshold. The same eval logic is exposed over MCP
so an agent can run evals as a tool — the part few portfolios show.

## Metrics

| Metric | Question it answers |
|--------|--------------------|
| **faithfulness** | Is the answer supported by the retrieved context? |
| **answer_relevancy** | Does the answer actually address the question? |
| **context_precision** | How much of the retrieved context was relevant? |

Two backends: a self-contained **Gemini LLM-as-a-Judge** (default), or
**RAGAS/DeepEval** (set `USE_RAGAS=true`).

## Quickstart

### 1. Install
```bash
git clone https://github.com/SaiArja/llm-eval-mcp.git
cd llm-eval-mcp
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
```

### 2. Configure
```bash
cp .env.example .env
# set GEMINI_API_KEY=...   (https://aistudio.google.com/app/apikey)
```

### 3. Run the eval gate
```bash
python -m scripts.run_eval
```
The bundled `data/sample_eval.jsonl` contains three rows — one is a deliberately
off-topic answer, so the aggregate drops the faithfulness/relevancy scores and
the gate **fails** (exit code 1). Remove that row, or raise answer quality, and
the gate **passes**. That pass/fail flip is exactly what runs in CI.

Get raw JSON:
```bash
python -m scripts.run_eval --json
```

## Running as an MCP server

```bash
python -m app.mcp_server.server
```

This starts an MCP server over stdio exposing three tools:

| Tool | Description |
|------|-------------|
| `run_eval(dataset_path)` | Full eval report + pass/fail gate |
| `score_sample(question, answer, contexts, ground_truth)` | Score one output |
| `get_thresholds()` | Current CI gate thresholds |

Register it with any MCP client (e.g. Claude Desktop) by pointing the client at
`python -m app.mcp_server.server`. An assistant can then evaluate RAG outputs on
demand as a tool call.

## CI gating (GitHub Actions)

`.github/workflows/eval.yml` runs unit tests and the eval gate on every push and
PR. Add your key as a repo secret named `GEMINI_API_KEY`
(Settings → Secrets and variables → Actions). A metric below threshold fails the
build — turning "did this change hurt retrieval quality?" into a red/green check.

## Dataset format

JSONL, one object per line:
```json
{"question": "...", "answer": "...", "contexts": ["...", "..."], "ground_truth": "..."}
```

## Project layout

```
app/
  eval/config.py     thresholds + backend selection
  eval/judge.py      Gemini LLM-as-a-Judge client
  eval/metrics.py    metric scoring (judge + optional RAGAS)
  eval/dataset.py    JSONL loader
  eval/runner.py     load -> score -> aggregate -> gate
  mcp_server/server.py  MCP tools
scripts/run_eval.py  CLI gate (exits non-zero on failure)
data/sample_eval.jsonl
.github/workflows/eval.yml
tests/               smoke tests (no key needed)
```

## Tests
```bash
pip install pytest
pytest -q
```

## License
MIT