# claude-code-eval-mcp

**A Model Context Protocol (MCP) server that lets Claude Code score its own tool-calling transcripts (or a batch of them) for hallucinated action claims, unsafe edit ordering, redundant tool-call loops, and pass@k across repeated attempts at a task.**

[![tests](https://github.com/futurerichdad/claude-code-eval-mcp/actions/workflows/tests.yml/badge.svg)](https://github.com/futurerichdad/claude-code-eval-mcp/actions)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)]()

## Why this exists

Most agent evaluation happens outside the agent's own conversation, as a separate offline pipeline you run after the fact. This server closes that gap for one specific, useful case: it puts transcript scoring directly inside a Claude Code (or Claude Desktop) session as a set of tools, so you can ask "score this run" or "what's the pass rate on this task" in plain language, mid-conversation, without leaving to run a separate script.

It is a thin wrapper. All the actual scoring logic (the rule checks, the pass@k aggregation) lives in the separate, independently public [apex-overlay](https://github.com/futurerichdad/apex-overlay) project, installed here as a real pip dependency, not copied in. A fix to a rule there is picked up here on the next `pip install --upgrade`, with zero duplicated code between the two repos. This project exists to demonstrate that composition: a focused MCP server built on top of an evaluation library, rather than one monolith trying to do both jobs.

## What it does

| Tool | Purpose |
|---|---|
| `list_violation_rules` | Documents the rule checks this scorer applies, so you know what a "violation" means before reading one |
| `score_transcript` | Scores a single transcript (passed as a JSON string) for rule violations |
| `score_transcript_file` | Scores every transcript in a JSONL file, returns aggregate metrics and pass@k |
| `get_pass_at_k` | Pass rate across repeated attempts, for one task_id or every task_id in a file |

The rules themselves (from apex-overlay's `harness/tool_call_rules.py`): `edited_without_reading` (a file modified with no prior Read of it in the transcript), `hallucinated_write_claim` / `hallucinated_read_claim` / `hallucinated_test_claim` (the model's own stated claims about what it did, checked against what tool calls actually appear in the transcript), and `redundant_tool_calls` (the same tool called three or more times consecutively with identical input).

## Data

Transcripts are JSONL, one tool-calling session per line: `session_id`, `task_id`, an ordered list of `steps` (each with a `tool`, its `input`, and the model's own `claimed_actions` text for that step), and a `final_outcome`. Full schema and a worked example: [apex-overlay's tool_call_schema.md](https://github.com/futurerichdad/apex-overlay/blob/main/examples/tool_call_schema.md). Any system that can export logs in this shape, or a small adapter that reshapes them into it, can be scored without touching this server's code.

## Quickstart

```bash
git clone https://github.com/futurerichdad/claude-code-eval-mcp.git
cd claude-code-eval-mcp
pip install -r requirements.txt

# Run the test suite
pytest tests/ -v

# Run the server (stdio transport)
python server.py
```

## Connecting to Claude Code

Add to your Claude Code MCP config (`claude mcp add` or your `mcp.json`):

```json
{
  "mcpServers": {
    "claude-code-eval": {
      "command": "python",
      "args": ["/absolute/path/to/claude-code-eval-mcp/server.py"]
    }
  }
}
```

Then try the prompts in [`examples/example_prompts.md`](examples/example_prompts.md). To try it against real data first, clone [apex-overlay](https://github.com/futurerichdad/apex-overlay) alongside this repo and generate its synthetic sample transcripts:

```bash
git clone https://github.com/futurerichdad/apex-overlay.git
cd apex-overlay
python data/generate_sample_transcripts.py --out sample_transcripts.jsonl --count 200
```

Then point any prompt at that file's absolute path, e.g. "score every transcript in /path/to/apex-overlay/sample_transcripts.jsonl and tell me the top violation."

## Architecture

```
Claude Code  --MCP (stdio)-->  server.py (MCPServer, 4 tools)
                                     |
                                     v
                    harness.tool_call_rules / tool_call_metrics
                    (installed from github.com/futurerichdad/apex-overlay)
                                     |
                                     v
                          transcripts.jsonl (caller-supplied)
```

## Design philosophy

- **Composition over duplication.** This server owns zero scoring logic. It owns the MCP tool contracts, input validation, and error shaping; apex-overlay owns the rules. Two focused public repos instead of one that does both jobs.
- **Structured, inspectable output.** Every tool returns JSON, matching apex-overlay's own report structure, so the numbers here match what `python -m harness.tool_call_harness` would print offline.
- **Errors the model can act on.** A missing file or unknown task_id returns `{"error": ..., "hint": ...}`, including the list of valid task_ids when relevant, so an agent can self-correct instead of dead-ending.

## Testing

```bash
pytest tests/ -v
```

11 tests covering all four tools: happy-path scoring, invalid/malformed JSON input, a missing file, an empty file, and an unknown task_id, both with and without an explicit `task_id` filter.

## About

Built by Ben, 9+ years of enterprise platform and fleet telemetry engineering (AT&T), now building agentic AI tooling. Anthropic Certified: Building with Claude API, Model Context Protocol. See also: [apex-overlay](https://github.com/futurerichdad/apex-overlay) (the evaluation harness this server wraps) and [fleet-data-mcp](https://github.com/futurerichdad/fleet-data-mcp) (an MCP server exposing fleet telemetry to Claude Code).

## License

MIT. See [LICENSE](LICENSE).