# mcp-retrieval-probe

An evaluation harness for MCP-based retrieval servers.

Retrieval servers that sit behind the Model Context Protocol are increasingly the
grounding layer for AI assistants: the assistant asks a question, the server
returns documents, and the assistant cites them. That makes one property more
important than raw relevance. **Does the server know when it does not know?**

A retrieval server that returns its best guess with a confident-looking score,
even when the corpus has nothing useful, will hand an assistant a citation it
should never have made. The model did not hallucinate. The grounding layer
manufactured the certainty and passed it up. This tool measures that, along with
the corpus and data-integrity properties that tend to drift quietly.

It is target-agnostic. Everything server-specific lives in a **recipe** file, so
the same probes run against any MCP retrieval server.

## Install

```bash
git clone https://github.com/borisknight/mcp-retrieval-probe
cd mcp-retrieval-probe
pip install -r requirements.txt
```

## Use

Write a recipe (start from `examples/example.recipe.json`), export your own API
token, and run a probe:

```bash
export MCP_TOKEN="your-own-token"
python -m mcp_retrieval_probe --recipe examples/example.recipe.json calibration
python -m mcp_retrieval_probe --recipe examples/example.recipe.json all --json
```

The token is read from the environment variable named in the recipe and is never
written to disk or logged. Use your own credentials against servers you are
permitted to test.

## The probes

| Probe | Question it answers |
|---|---|
| `calibration` | Does the confidence signal mean anything, and does the server ever abstain? |
| `anisotropy` | Do similarity scores have enough dynamic range to threshold on? |
| `relevance` | Are results on topic, and does a paper retrieve itself by title? |
| `coverage` | Does the corpus match its claims, and how much of it is embedded? |
| `citations` | Is citation data present and correct, or is "unknown" reported as zero? |
| `conservation` | Do two endpoints counting the same thing agree? |
| `discovery` | Does the OAuth discovery metadata match the documented endpoint? |

Each probe prints a one-line verdict and returns a structured result under
`--json`.

### Why these, specifically

**calibration** sends three buckets of queries: topics the corpus covers, real
scholarship it almost certainly does not, and pure gibberish. A calibrated server
looks visibly less certain on the last two. One that returns an identical
confidence label for all three is not giving its callers a signal at all.

**anisotropy** explains *why* a confidence label often fails. In an anisotropic
embedding space nearly every pair of vectors is highly cosine similar, so scores
bunch into a narrow band and no fixed threshold can separate a strong match from
a weak one. The fix is usually normalisation before thresholding, not a new
label.

**citations** separates two things servers routinely conflate: a document with
zero citations, and a document whose citation data failed to link. Reporting the
second as the first makes landmark work look ignored.

**discovery** needs no credentials. It compares the canonical resource identifier
a server advertises against the endpoint it documents, a mismatch that produces
confusing auth failures for spec-compliant clients.

## Recipes

A recipe has three sections:

- `transport` — `base_url`, `protocol_version`, and `token_env` (the name of the
  environment variable holding the bearer token).
- `tools` — maps each *logical* tool a probe needs to the *concrete* tool the
  server exposes. Each entry has a `name`, an `args` template using `$QUERY`,
  `$LIMIT`, `$ID`, `$CATEGORY`, `$START`, `$END`, `$SOURCE` placeholders, and a
  `fields` map of dotted response paths. A placeholder whose value is `None` is
  dropped from the call, so optional filters can be omitted. Logical tools the
  recipe leaves out simply cause the probes that need them to skip.
- `probes` — per-probe configuration: query buckets, category anchors, seeds,
  the OpenAlex concept to audit, and so on.

Ground truth for the citation probe comes from the public
[OpenAlex](https://openalex.org) API.

## Running it as an MCP server

The probes are also exposed as MCP tools, so an assistant can run an evaluation
in conversation and a CI agent can call them directly:

```bash
pip install 'mcp>=2.0'
```

```json
{
  "mcpServers": {
    "retrieval-probe": {
      "command": "python",
      "args": ["-m", "mcp_retrieval_probe.server"],
      "env": {
        "MCP_PROBE_RECIPE": "/path/to/recipe.json",
        "MCP_TOKEN": "your-own-token"
      }
    }
  }
}
```

Each probe becomes a `probe_<name>` tool, plus `probe_all`.

## Tests

```bash
pip install -r requirements-dev.txt
pytest
```

The default suite is offline. It needs no network and no credentials, and it
covers three layers:

- **Transport** — the MCP handshake, session id handling, SSE parsing, payload
  unwrapping, and the 401 and JSON-RPC error paths, against a stubbed HTTP call.
- **Probe logic** — each probe is shown to fire on a server with the defect it
  looks for and stay quiet on one without it. A verdict is only worth reporting
  if you can demonstrate it discriminates.
- **CLI** — the paths a first-time user actually hits: a missing token, a bad
  recipe path, `--json` output, and a probe that raises without aborting the run.

Functional checks against a real server are opt-in, so CI stays credential-free:

```bash
export MCP_PROBE_RECIPE=/path/to/recipe.json
export MCP_TOKEN=...          # whatever token_env your recipe names
pytest -m live
```

Those assert reachability and response shape rather than specific findings,
since findings depend on the server under test.

## License

MIT. See [LICENSE](LICENSE).