evalgate-mcp

An MCP server that lets an agent read its own eval results. Wraps evalgate — run a suite, explain a failure, check for drift, and ask whether the judge was ever calibrated.

Phase 1 is read-only. Nothing here writes a file. run_suite executes the system under test and may bill a provider, but the result comes back in memory rather than landing in .evalgate/result.json. Write access is phase 2, deliberately.

Why an agent should be allowed to do this

Handing an agent its own grader is circular. It's also useful, and the circularity is manageable as long as one thing is never hidden: whether anyone measured the grader.

So every result from run_suite carries a judgeStatus block, in one of five states:

That block is the point of this server. An agent that can score its own output and can't see the state of its grader will report success it didn't earn.

Related MCP server: mcp-llm-eval

Tools

Bad paths and malformed suites come back as tool errors with the message intact, not as thrown stack traces the model can't act on.

Install

Not on npm. evalgate isn't either — both install from git.

git clone https://github.com/HolisticShort/evalgate-mcp.git
cd evalgate-mcp
npm install          # builds on install via `prepare`
npm run smoke        # end-to-end check against evalgate's example suite

npm run smoke expects an evalgate checkout at ../evalgate, or set EVALGATE_DIR.

Use it with Claude Code

Add to .mcp.json in the project you want to gate:

{
  "mcpServers": {
    "evalgate": {
      "command": "node",
      "args": ["/absolute/path/to/evalgate-mcp/dist/server.js"]
    }
  }
}

Restart Claude Code, then /mcp to confirm the server connected and the five tools are listed.

Phase 2

Write access — letting an agent add cases, adjust thresholds, and write baselines. Held back on purpose. An agent editing the test cases that grade it is a different trust question from an agent reading them, and it deserves its own design rather than arriving as a convenience.

License

MIT