# evalgate-mcp

**An MCP server that lets an agent read its own eval results.** Wraps
[evalgate](https://github.com/HolisticShort/evalgate) — run a suite, explain a failure, check for drift,
and ask whether the judge was ever calibrated.

> **Phase 1 is read-only.** Nothing here writes a file. `run_suite` executes the system under test and may
> bill a provider, but the result comes back in memory rather than landing in `.evalgate/result.json`.
> Write access is phase 2, deliberately.

---

## Why an agent should be allowed to do this

Handing an agent its own grader is circular. It's also useful, and the circularity is manageable as long as
one thing is never hidden: **whether anyone measured the grader.**

So every result from `run_suite` carries a `judgeStatus` block, in one of five states:

| state | meaning |
|---|---|
| `no-judge` | No judged assertions ran. Nothing to qualify. |
| `calibrated` | The judge has a passing stamp bound to its own id. Agreement is published. |
| `uncalibrated` | No stamp exists. Scores are unqualified — a judge nobody measured is a random number generator with good manners. |
| `judge-mismatch` | A stamp exists but belongs to a different judge. Agreement is withheld, because one judge inheriting another's credibility is worse than no stamp at all. |
| `failed-calibration` | The judge last failed calibration. Agreement is reported *with* the failure. |

That block is the point of this server. An agent that can score its own output and can't see the state of
its grader will report success it didn't earn.

## Tools

| tool | cost | what it does |
|---|---|---|
| `list_suites` | free | Load and validate suites — cases, assertions, thresholds. |
| `explain_result` | free | Read a result artifact and explain what failed, down to the claim-level grounding breakdown. |
| `check_drift` | free | Analyze `history.jsonl` for slow decline the per-PR gate can't see. |
| `judge_status` | free | Read the calibration stamp and report agreement, bias state, and judge binding. |
| `run_suite` | **executes the SUT; may call providers** | Run the suites and return the scored result. Writes nothing. |

Bad paths and malformed suites come back as tool errors with the message intact, not as thrown stack traces
the model can't act on.

## Install

Not on npm. evalgate isn't either — both install from git.

```bash
git clone https://github.com/HolisticShort/evalgate-mcp.git
cd evalgate-mcp
npm install          # builds on install via `prepare`
npm run smoke        # end-to-end check against evalgate's example suite
```

`npm run smoke` expects an evalgate checkout at `../evalgate`, or set `EVALGATE_DIR`.

## Use it with Claude Code

Add to `.mcp.json` in the project you want to gate:

```json
{
  "mcpServers": {
    "evalgate": {
      "command": "node",
      "args": ["/absolute/path/to/evalgate-mcp/dist/server.js"]
    }
  }
}
```

Restart Claude Code, then `/mcp` to confirm the server connected and the five tools are listed.

## Phase 2

Write access — letting an agent add cases, adjust thresholds, and write baselines. Held back on purpose.
An agent editing the test cases that grade it is a different trust question from an agent reading them, and
it deserves its own design rather than arriving as a convenience.

## License

MIT