# modelrefs-mcp

An MCP server for the [ModelRefs Evidence API](https://modelrefs.com/documentation#mcp). It gives
an agent governed benchmark evidence, provider-documented retirement dates, and comparisons that
refuse to rank scores that are not comparable — inside the editor, where the model choice is
actually being made.

It holds no data of its own. It wraps three HTTP endpoints and is published separately so that
installing it does not mean trusting a bundled copy of anything.

## Install

Claude Desktop, Claude Code, Cursor, Windsurf and anything else that reads an `mcpServers` block:

```json
{
  "mcpServers": {
    "modelrefs": {
      "command": "npx",
      "args": ["-y", "modelrefs-mcp"],
      "env": { "MODELREFS_API_KEY": "mr_live_…" }
    }
  }
}
```

Claude Code, in one line:

```bash
claude mcp add modelrefs --env MODELREFS_API_KEY=mr_live_… -- npx -y modelrefs-mcp
```

The key is optional. Without it the server still starts and `list_covered_models` still answers,
which is the tool worth trying first. Keys are created at
[modelrefs.com/api-keys](https://modelrefs.com/api-keys); the free tier includes 1,000 calls a
month. The key is read from the environment and never from a command-line argument, because argv
is visible to every process on the machine.

> **Don't pin the version in `args` on Windows.** `npx -y modelrefs-mcp` works;
> `npx -y modelrefs-mcp@<any-version>` fails with `'modelrefs-mcp' is not recognized as an
> internal or external command` when launched by an MCP client. The package is identical either
> way — the same build runs fine through its own `.bin` shim and through `npx --no-install` — so
> this is npx resolving a bin from its cache when it is given a version spec, not something in
> the server. If you need a pinned version, install it and name the binary directly:
>
> ```json
> { "command": "modelrefs-mcp", "args": [] }
> ```
>
> after `npm i -g modelrefs-mcp@<version>`.

## Tools

| Tool | Needs a key | What it answers |
| --- | --- | --- |
| `list_covered_models` | No | Which model slugs ModelRefs holds governed evidence for. |
| `get_model_evidence` | Yes | A model's citations and governed benchmark runs, each with its source, claim scope and the methodology its publisher did **not** disclose. |
| `check_model_lifecycle` | Yes | Whether a model is deprecated, when it retires and what replaces it — resolved for the snapshot your code actually pins. |
| `compare_models` | Yes | Two to five models on the benchmarks where their runs are genuinely comparable, plus everything that could not be compared and why. |

### Why `compare_models` refuses things

Two runs are put side by side only when the **benchmark**, the **metric** and the **task subset**
all match. A shared benchmark name is not comparability.

The case that forced the rule: Anthropic reports tau-bench Retail and Airline separately, and the
same model differs by roughly twenty points between them, while OpenAI reports tau2-bench telecom.
Printing 96.7 on telecom beside 81.4 on Retail reads as a capability gap and is mostly a domain
difference. So those land in an `incomparable` list with the reason, rather than being ranked.

Benchmarks only one model has evidence for land in `oneSided`, because "A beats B" and "we only
measured A" are different claims.

### Why `check_model_lifecycle` wants a snapshot id

Providers retire individual snapshots months before the family, with a different replacement.
`gpt-4o` retires 2027-01-14; `gpt-4o-2024-05-13` retired 2026-10-23 and pointed somewhere else. A
family-level answer is wrong by months for whoever pinned early, so pass the dated id your code
actually uses:

```
check_model_lifecycle(slug: "gpt-4o", snapshot: "gpt-4o-2024-05-13")
```

With no snapshot, the earliest date across the family and all its snapshots is returned — never
the family date alone. If you pass a snapshot there is no record for, the answer says so rather
than quietly giving you the family date.

## What it will not do

- **It does not run inference.** No completion, embedding or image tools. This is a reference
  layer, not a model gateway.
- **It does not return a score without its scope.** Every run carries `claimScope` and
  `methodologyLimitations`; the tool descriptions tell the agent not to quote one without them.
- **It does not pretend to cover the catalogue.** ModelRefs holds governed evidence for a subset.
  A model outside it answers 404 rather than an empty payload, because an empty payload invites a
  reader to treat absence as a finding. Call `list_covered_models` first.
- **It does not infer a retirement date.** A model with no reviewed lifecycle record answers 404,
  and that means *unknown*, not *active*.
- **It does not cache.** A stale retirement date is the one answer this product must never give,
  and a cache in a long-lived editor session is exactly how you give one.

## Development

```bash
npm install
npm run build
npm run selftest
```

This repository is a mirror. The package is developed inside the ModelRefs
monorepo, where `src/generated.ts` is produced from the same registries that
publish the website and the Evidence API, and a governance gate fails the build
if the copy here and the source there ever disagree. That is the point of the
arrangement: the tool table this server exposes and the endpoint list the API
documents cannot drift apart, because they are one table.

What that means for you: pull requests here are welcome and are applied
upstream, but a fix lands in the monorepo first and arrives in this repository
with the release that carries it. For anything else — a wrong retirement date, a
model you need covered, a bug in an answer — <https://modelrefs.com/contact>
reaches someone faster than an issue here.

Three checks, each proving something the others cannot:

| Script | What only it can catch |
| --- | --- |
| `npm run selftest` | The server's own logic, against a stub serving the live shapes — including the awkward cases the live data does not currently contain. |
| `npm run verify:tarball` | A broken `files` list, `bin` entry, shebang or module resolution in the packed artifact — things that fail only after someone installs it. |
| `npm run verify:registry` | That what npm actually **serves** resolves, downloads and runs. Run it after publishing. |

The self-test connects a real MCP client to the server over an in-memory transport and points it
at a local stub of the Evidence API, so what it exercises is the server that ships. It asserts the
refusals, not only the answers: a suite that only checked the happy path would pass against a
version that ranked tau-bench Retail against tau2-bench telecom, which is the one defect this code
exists to prevent.

The tool table is generated from `src/lib/api-docs/mcp.ts` in the main repository, which the
reference page also renders, and `npm run governance:mcp-parity` there fails if the copy drifts or
a tool names an endpoint that does not exist.

## Changelog

**0.1.2** — Registry metadata only; the server code is unchanged from 0.1.0. Moves the MCP
registry name from the GitHub namespace to the domain namespace, `com.modelrefs/evidence`, which
is the one ModelRefs can prove it owns, and adds the `mcpName` field that 0.1.0 and 0.1.1 both
shipped without — the registry reads it from the published package to confirm the package and the
server entry are the same project, and refuses the submission when it is absent.

**0.1.1** — Documentation only; the server code is unchanged from 0.1.0. Adds the Windows note
above, after `npx -y modelrefs-mcp@<version>` was found to fail under the launcher MCP clients
use while the unversioned form works. 0.1.0 shipped before that was known, so its README does
not warn about it.

**0.1.0** — First release. Four tools over three Evidence API endpoints.

## Licence

MIT.