# MCP Eval Demo

A worked example of using **evaluations to verify that an LLM agent can actually use an
MCP server effectively** — not just that the server's code is correct.

Unit tests answer "does `delete_note` delete a note?" They cannot answer the questions
that decide whether an MCP server is any good in practice:

- Does the agent find the right note when the user describes it in words instead of by id?
- Does it notice that a listing preview was truncated, or does it answer from half a note?
- Does it realize `update_note` overwrites, or does it silently destroy the user's content
  when asked to "add a line to my grocery list"?
- Does it recover from an error message, or give up?

Those are properties of the **tool surface** — names, descriptions, schemas, result shapes,
error text — and the only way to check them is to run a real agent against the server and
grade what it did. That is what this repo is for.

## Status

The MCP server, its infrastructure, and the eval harness are all in place.

## The server under test: Notes MCP

An in-memory notebook. State lives in the server process and is discarded on exit, so every
eval run starts from the same known corpus (see [seed.py](src/notes_mcp/seed.py)).

| Tool | Behaviour hint | What it does |
| --- | --- | --- |
| `create_note` | write | Creates a note; titles must be unique case-insensitively. |
| `get_note` | read-only | Returns one note's **full** content, by id. |
| `list_notes` | read-only | Lists notes newest-updated first, as truncated previews, with an optional substring `query`. |
| `update_note` | destructive | **Overwrites** the title and/or content of a note. |
| `delete_note` | destructive | Permanently removes a note. |

Several design choices exist specifically to give the evals something to catch:

- **Ids, not titles.** Every mutating tool takes a `note_id`, so an agent asked to change
  "my grocery list" must look the id up first. This is where agents commonly guess.
- **Truncated previews.** `list_notes` returns only the first 120 characters of each note,
  flagged with `content_truncated` and `content_length`. An agent that answers a
  content question straight from a listing gets it wrong; a good one calls `get_note`.
- **Replace, not append.** `update_note` overwrites. "Add eggs to my grocery list" is
  therefore a read-modify-write, and an agent that skips the read destroys data.
- **Errors that teach.** Every failure names the offending value and points at the tool that
  would resolve it, so an agent has a path forward rather than a dead end.

## Layout

```
src/notes_mcp/
  models.py    Pydantic models — also the tool input/output schemas the agent sees
  store.py     In-memory storage and its error types
  seed.py      Fixed corpus: stable ids and timestamps, so evals are reproducible
  server.py    MCP tool definitions, descriptions, and annotations
  cli.py       `notes-mcp` entry point
evals/
  agent.py       Builds the pydantic-ai agent under test + local trace capture
  task.py        One agent turn against a freshly seeded server — the thing evaluated
  evaluators.py  Custom pydantic-evals evaluators (tool-not-called, argument-contains)
  cases.yaml     The dataset itself: cases that probe specific MCP misuse patterns
  cases.py       Loads cases.yaml — registers the custom evaluators, picks the judge model
  __main__.py    `python -m evals` — runs the dataset against a live model
tests/
  test_store.py    Unit tests for the storage layer
  test_server.py   Protocol-level tests through a real MCP client session
scripts/
  lint.sh    Ruff + pyright + format check
  test.sh    Unit + protocol tests (fast, free)
  evals.sh   Agent-behaviour evals against a live model (slow, costs money)
```

The tool descriptions are module-level constants in
[server.py](src/notes_mcp/server.py) rather than inline docstrings. Description wording is
the main thing you tune in response to a failing eval, and keeping it in one place makes
those diffs readable.

## Getting started

Requires [uv](https://docs.astral.sh/uv/) and Python 3.12 (pinned in
[.python-version](.python-version)).

```bash
uv sync                       # create .venv and install everything
uv run scripts/test.sh        # unit + protocol tests
uv run scripts/lint.sh        # ruff check, pyright (strict), format check
uv run pre-commit install     # optional: run the same checks on commit
```

### Running the server

```bash
uv run notes-mcp                        # stdio, seeded with the sample notes
uv run notes-mcp --empty                # stdio, no notes
uv run notes-mcp --transport streamable-http
```

[.mcp.json](.mcp.json) registers the stdio server for this project, so an MCP host
launched from this directory — Claude Code, for instance — picks up the `notes` server
automatically and you can drive it by hand.

To read the tool surface an agent would see — the thing these evals are really about —
without starting an agent at all:

```bash
uv run fastmcp list .mcp.json                   # names, signatures, descriptions
uv run fastmcp list .mcp.json --input-schema    # ...with the full JSON schemas
npx @modelcontextprotocol/inspector uv run notes-mcp   # MCP Inspector, for clicking around
```

> **On the `mcp` version:** the server is built on the standalone
> [FastMCP](https://gofastmcp.com) library rather than the copy that used to ship inside
> the `mcp` SDK as `mcp.server.fastmcp` — mcp 2.0 removed that module. FastMCP owns which
> `mcp` version it needs (3.x resolves mcp 1.x), so [pyproject.toml](pyproject.toml) has no
> hand-written `mcp` bound. The eval harness arrives at the same library from the other
> side: `pydantic-ai`'s MCP client is built on FastMCP's `Client`. Both halves of this repo
> therefore agree on a version by construction rather than by a pin someone has to
> maintain. FastMCP 4 is the step that moves both to mcp 2.x, which is why the dependency
> is capped below it.

## Testing approach

Two pytest layers, run by `scripts/test.sh`:

- **`test_store.py`** covers storage semantics — uniqueness, sort order, limits, timestamps.
  Fast, exhaustive, no protocol involved.
- **`test_server.py`** drives the server through an in-process MCP client session
  (`fastmcp.Client`, over FastMCP's in-memory transport), so it asserts on what an agent
  actually receives: the tool list, JSON schemas, behaviour annotations, structured
  results, and error text. The protocol is real; only the subprocess and socket are not.

Async tests use the `anyio` pytest plugin rather than `pytest-asyncio`, because the MCP
client holds a cancel scope open for the life of the session and anyio runs fixture setup
and teardown in the same task.

A third kind of check — the agent-behaviour evals — calls a live model and costs money,
so it isn't part of the pytest suite at all; it has its own runner and script, described
next.

## The eval harness

[`evals/`](evals/) builds a minimal [pydantic-ai](https://ai.pydantic.dev/) agent — a
generic one-line system prompt, no few-shot examples, no special-cased instructions — and
wires its *only* tools to an in-process Notes MCP server via `pydantic_ai.mcp.MCPToolset`
([agent.py](evals/agent.py)). The system prompt is deliberately bare: these evals exist to
check whether the server's own tool names, descriptions, and schemas are enough to guide
correct behaviour, not whether prompt engineering can paper over a weak one.

`evals/` lives at the top level rather than under `src/`: it's dev tooling for this repo,
not part of the `notes-mcp` package anyone would install.

[`pydantic_evals`](https://ai.pydantic.dev/evals/) runs that agent over a `Dataset` of
`Case`s, each targeting one of the four behaviours from the top of this file:

| Case | Checks |
| --- | --- |
| `delete_by_description_looks_up_the_id_first` | Asked to delete "my grocery list note," the agent calls `list_notes` before `delete_note`, and deletes the *right* id. |
| `answers_past_the_list_notes_preview_cutoff` | A question whose answer is truncated out of the `list_notes` preview is only answered correctly if the agent calls `get_note`. |
| `appending_to_a_note_preserves_its_truncated_tail` | "Add crackers to my grocery list" must read the full note first — the `update_note` call is checked for text that only exists past the preview cutoff. |
| `title_conflict_on_create_is_not_silently_lost` | Creating a note whose title already exists must not silently drop the new content or claim a duplicate was created. |
| `deleting_a_nonexistent_note_does_not_fabricate_success` | A request to delete a note that doesn't exist must not result in a `delete_note` call with a guessed id, or a reply claiming success. |
| `simple_lookup_answers_from_the_right_note` | A sanity-check happy path. |

The cases live in [cases.yaml](evals/cases.yaml) rather than in Python — they're data, so
adding a case or rewording a rubric doesn't touch code. [cases.py](evals/cases.py) is only
the loader: it hands the custom evaluators to `Dataset.from_file` (a YAML file can only
name an evaluator the loader registers) and sets the judge model. The YAML's
`yaml-language-server` header points at [cases_schema.json](evals/cases_schema.json), so an
editor can complete and validate evaluator names and their arguments; regenerate it after
adding or changing a custom evaluator:

```bash
uv run python -c "from evals.cases import write_json_schema; print(write_json_schema())"
```

Evaluators combine pydantic-evals' built-ins (`ToolCorrectness`, `Contains`, `MaxToolCalls`,
`LLMJudge` for the two cases with more than one valid recovery) with two small custom ones
in [evaluators.py](evals/evaluators.py): `ToolNotCalled` (assert a tool was never invoked —
there's no built-in negative check) and `ArgumentContains` (substring checks on a tool
argument, for "the old content must survive" cases where an LLM's exact wording can't be
pinned down with an equality or subset-dict match). Both, like the built-ins, read
tool-call spans that `Agent.instrument_all()` plus a local (`send_to_logfire=False`)
`logfire.configure()` capture — see `configure_instrumentation()` in
[agent.py](evals/agent.py).

[`__main__.py`](evals/__main__.py) runs the dataset, prints a full report, and exits
non-zero if anything failed — a task error, a crashed evaluator, or a failed assertion.
Run it with:

```bash
uv run scripts/evals.sh
```

### Configuring a provider

`NOTES_MCP_EVAL_MODEL` picks both the provider and model, as a pydantic-ai
`provider:model` string, and defaults to `anthropic:claude-haiku-4-5-20251001`. Copy
[.env.example](.env.example) to `.env` and fill in the section for whichever of these
three you're using — `scripts/evals.sh` loads `.env` automatically (via `python-dotenv`;
it never overrides a variable already set in your shell) and `.env` is gitignored:

- **Anthropic API** (default) — needs `ANTHROPIC_API_KEY`.
- **OpenAI** — `NOTES_MCP_EVAL_MODEL=openai:gpt-5` and `OPENAI_API_KEY`.
- **Amazon Bedrock** — `NOTES_MCP_EVAL_MODEL=bedrock:<bedrock-model-id>`. Authenticates
  through boto3's normal credential chain, so there's nothing eval-specific to
  configure beyond the standard AWS SDK variables: set `AWS_PROFILE` to use a named
  profile (`AWS_DEFAULT_REGION` too, if that profile doesn't already set a region — note
  it must be `AWS_DEFAULT_REGION`, not `AWS_REGION`, which boto3's region resolution
  doesn't check), or leave both unset to use your default profile/region.

No code branches on the provider — `eval_model()`'s string is handed straight to both
the agent and `LLMJudge`, and pydantic-ai's `infer_model` resolves the right client and
credentials for whichever provider prefix it sees.