claude-code-eval-mcp

A Model Context Protocol (MCP) server that lets Claude Code score its own tool-calling transcripts (or a batch of them) for hallucinated action claims, unsafe edit ordering, redundant tool-call loops, and pass@k across repeated attempts at a task.

Why this exists

Most agent evaluation happens outside the agent's own conversation, as a separate offline pipeline you run after the fact. This server closes that gap for one specific, useful case: it puts transcript scoring directly inside a Claude Code (or Claude Desktop) session as a set of tools, so you can ask "score this run" or "what's the pass rate on this task" in plain language, mid-conversation, without leaving to run a separate script.

It is a thin wrapper. All the actual scoring logic (the rule checks, the pass@k aggregation) lives in the separate, independently public apex-overlay project, installed here as a real pip dependency, not copied in. A fix to a rule there is picked up here on the next pip install --upgrade, with zero duplicated code between the two repos. This project exists to demonstrate that composition: a focused MCP server built on top of an evaluation library, rather than one monolith trying to do both jobs.

Related MCP server: Petamind MCP

What it does

The rules themselves (from apex-overlay's harness/tool_call_rules.py): edited_without_reading (a file modified with no prior Read of it in the transcript), hallucinated_write_claim / hallucinated_read_claim / hallucinated_test_claim (the model's own stated claims about what it did, checked against what tool calls actually appear in the transcript), and redundant_tool_calls (the same tool called three or more times consecutively with identical input).

Data

Transcripts are JSONL, one tool-calling session per line: session_id, task_id, an ordered list of steps (each with a tool, its input, and the model's own claimed_actions text for that step), and a final_outcome. Full schema and a worked example: apex-overlay's tool_call_schema.md. Any system that can export logs in this shape, or a small adapter that reshapes them into it, can be scored without touching this server's code.

Quickstart

git clone https://github.com/futurerichdad/claude-code-eval-mcp.git
cd claude-code-eval-mcp
pip install -r requirements.txt

# Run the test suite
pytest tests/ -v

# Run the server (stdio transport)
python server.py

Connecting to Claude Code

Add to your Claude Code MCP config (claude mcp add or your mcp.json):

{
  "mcpServers": {
    "claude-code-eval": {
      "command": "python",
      "args": ["/absolute/path/to/claude-code-eval-mcp/server.py"]
    }
  }
}

Then try the prompts in examples/example_prompts.md. To try it against real data first, clone apex-overlay alongside this repo and generate its synthetic sample transcripts:

git clone https://github.com/futurerichdad/apex-overlay.git
cd apex-overlay
python data/generate_sample_transcripts.py --out sample_transcripts.jsonl --count 200

Then point any prompt at that file's absolute path, e.g. "score every transcript in /path/to/apex-overlay/sample_transcripts.jsonl and tell me the top violation."

Architecture

Claude Code  --MCP (stdio)-->  server.py (MCPServer, 4 tools)
                                     |
                                     v
                    harness.tool_call_rules / tool_call_metrics
                    (installed from github.com/futurerichdad/apex-overlay)
                                     |
                                     v
                          transcripts.jsonl (caller-supplied)

Design philosophy

• Composition over duplication. This server owns zero scoring logic. It owns the MCP tool contracts, input validation, and error shaping; apex-overlay owns the rules. Two focused public repos instead of one that does both jobs.

• Structured, inspectable output. Every tool returns JSON, matching apex-overlay's own report structure, so the numbers here match what python -m harness.tool_call_harness would print offline.

• Errors the model can act on. A missing file or unknown task_id returns {"error": ..., "hint": ...}, including the list of valid task_ids when relevant, so an agent can self-correct instead of dead-ending.

Testing

pytest tests/ -v

11 tests covering all four tools: happy-path scoring, invalid/malformed JSON input, a missing file, an empty file, and an unknown task_id, both with and without an explicit task_id filter.

About

Built by Ben, 9+ years of enterprise platform and fleet telemetry engineering (AT&T), now building agentic AI tooling. Anthropic Certified: Building with Claude API, Model Context Protocol. See also: apex-overlay (the evaluation harness this server wraps) and fleet-data-mcp (an MCP server exposing fleet telemetry to Claude Code).

License

MIT. See LICENSE.