# multivon-mcp

[![PyPI](https://img.shields.io/pypi/v/multivon-mcp.svg)](https://pypi.org/project/multivon-mcp)
[![Python](https://img.shields.io/badge/python-3.10+-blue.svg)](https://pypi.org/project/multivon-mcp)
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
[![Downloads](https://static.pepy.tech/badge/multivon-mcp/month)](https://pepy.tech/project/multivon-mcp)

**[Docs](https://docs.multivon.ai/mcp)** · [Website](https://multivon.ai/agents) · [PyPI](https://pypi.org/project/multivon-mcp) · [multivon-eval (engine)](https://github.com/multivon-ai/multivon-eval) · [Changelog](CHANGELOG.md)

These 23 tools cover what an autonomous eval agent needs to do its job: discover its own capabilities (`eval_discover`), normalize traces from supported sources (`eval_ingest_trace`), and run evaluators against them, with calibration evidence specific to the tested task. We put the framework behind an MCP boundary because eval belongs in the agent's working loop, not behind a separate dashboard.

An MCP server that gives AI coding agents direct access to evaluation tools. Drop into Claude Desktop, Claude Code, Cursor, Cline, or any [Model Context Protocol](https://modelcontextprotocol.io/)–compatible agent.

When the agent is helping you build an LLM product, it can:

- Score a RAG output for hallucination without you writing the scaffolding
- Generate an adversarial PDF on demand to test your document AI
- Run the full pdfhell mini-suite against a model and analyse the results
- Produce a self-verifying audit pack with a SHA-256 file manifest
- Discover the full evaluation capability catalog as JSON

No copy-paste, and no asking the agent to figure out the SDK calls from `python -c "..."` one-liners.

> **Current release: 0.4.0.** Tested with MCP Python SDK 1.29.x, multivon-eval 0.18.0, and pdfhell 0.6.2. See the [changelog](CHANGELOG.md).

## Install

```bash
pip install "multivon-mcp==0.4.0"
```

The next release carries this compatibility bound itself. Installation pulls
`multivon-eval`, `pdfhell`, and the MCP SDK. The provider SDKs (`anthropic`,
`openai`, `google-genai`) come along too — bring your own API key in env.

## Configure your agent

### Claude Code

```bash
claude mcp add --transport stdio --scope user multivon -- multivon-mcp
claude mcp get multivon
```

Set provider keys in your shell or secure environment before starting Claude Code. To share the server configuration with a project instead, use `--scope project`; Claude Code writes `.mcp.json` and supports environment-variable expansion there. It does **not** read `claude_desktop_config.json`.

### Claude Desktop

Add to `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows):

```json
{
  "mcpServers": {
    "multivon": {
      "command": "multivon-mcp",
      "env": {
        "ANTHROPIC_API_KEY": "sk-ant-...",
        "OPENAI_API_KEY": "sk-proj-...",
        "GOOGLE_API_KEY": "AIza..."
      }
    }
  }
}
```

Restart Claude. The 23 tools become available; ask Claude `"use multivon to evaluate this RAG output"` and it figures out which tool to call.

### Cursor

`.cursor/mcp.json` or via Settings → MCP:

```json
{ "mcpServers": { "multivon": { "command": "multivon-mcp" } } }
```

### Cline / OpenCode / any MCP-compatible agent

Same shape — point at the `multivon-mcp` console script.

### Local dev / debugging

From a clone of this repo:

```bash
mcp dev multivon_mcp/server.py
```

From a pip install (the file lives in site-packages, so resolve it):

```bash
mcp dev "$(python -c 'import multivon_mcp.server as s; print(s.__file__)')"
```

Opens the MCP Inspector UI in your browser. You can call any tool by name, see the JSON schemas, and watch the requests/responses.

## The 23 tools

### Discovery & document AI

| Tool | What it does | API key |
|---|---|---|
| `eval_discover` | Full machine-readable capability catalog (evaluators, traps, suites, calibration data, versions). Call first. | No |
| `pdfhell_make` | Generate one adversarial PDF + its answer key. | No |
| `pdfhell_run` | Run the pdfhell adversarial-PDF benchmark against a vision model. Returns pass rate, per-trap CIs, suite hash. | Yes (vision) |
| `eval_audit_pack` | Build a procurement-ready ZIP with a SHA-256 file manifest from a pdfhell run. | No |

### RAG generation & retrieval

| Tool | What it does | API key |
|---|---|---|
| `eval_faithfulness` | QAG-graded faithfulness — is a RAG output grounded in the retrieved context? | Yes |
| `eval_hallucination` | QAG-graded hallucination — does the output contain content NOT in context? | Yes |
| `eval_relevance` | QAG-graded answer-vs-question relevance. | Yes |
| `eval_answer_accuracy` | QAG-graded semantic equivalence vs ground truth. | Yes |
| `eval_context_precision` | RAG retrieval quality — are the retrieved chunks on-topic? | Yes |
| `eval_context_recall` | RAG retrieval completeness — does context contain enough info to answer? | Yes |

### Safety, compliance, fairness

| Tool | What it does | API key |
|---|---|---|
| `eval_toxicity` | QAG-graded toxicity / harmful-content detection. | Yes |
| `eval_bias` | QAG-graded bias across gender, race, politics, age, socioeconomic axes. | Yes |
| `eval_pii_detection` | Local-only regex scan for PII (GDPR / CCPA / PIPEDA / HIPAA / DPDP packs). | No |
| `eval_schema_compliance` | Validate an LLM output against a JSON Schema. | No |

### Agent & multimodal

| Tool | What it does | API key |
|---|---|---|
| `eval_tool_call_accuracy` | Deterministic agent tool-call correctness. No LLM. | No |
| `eval_vqa_faithfulness` | Image-grounded visual-QA faithfulness. | Yes (vision) |
| `eval_document_grounding` | Multi-page document-grounded faithfulness for document-AI agents. | Yes (vision) |

> **Agent traces.** `eval_tool_call_accuracy` and the other agent-trace
> evaluators in `multivon-eval` (`ToolArgumentAccuracy`,
> `ToolCallNecessity`, `TrajectoryEfficiency`, `AgentMemoryEval`,
> `PlanQuality`, `TaskCompletion`, `StepFaithfulness`) take an
> `agent_trace=[AgentStep(...)]` plus `expected_tool_calls=[...]` on
> the case. Three-shape semantics matter: `expected_tool_calls=None`
> skips, `[]` asserts "no tools called", and `[...]` checks for the
> named calls. On repository `main` (shipping in the next release), the MCP
> tool supports the same trace mode: normalize
> trace JSON with `eval_ingest_trace`, then pass its `agent_trace` plus
> `expected_tool_calls` to `eval_tool_call_accuracy`. Set
> `require_order=true` when sequence matters or
> `penalize_unexpected=true` for a strict allow-list. See the
> [`multivon-eval` agent integrations](https://github.com/multivon-ai/multivon-eval/tree/main/multivon_eval/integrations)
> for the source-of-truth tracer code.

### Flexible scoring

| Tool | What it does | API key |
|---|---|---|
| `eval_g_eval` | G-Eval holistic 0.0-1.0 scoring against a plain-English criterion. | Yes |
| `eval_custom_rubric` | Score against your own list of yes/no quality checks. | Yes |

### Agent workflows (new in 0.3.0)

| Tool | What it does | API key |
|---|---|---|
| `eval_acceptance_report` | Apply required-check, coverage and slice policy to a saved report; returns accept/reject/indeterminate. | No |
| `eval_compare_runs` | Diff two eval report JSONs — pass-rate delta, per-case regressions/improvements, McNemar p-value. Includes identity warnings; inconclusive comparisons cannot confirm improvement. | No |
| `eval_generate_cases` | Generate N eval cases (input / expected_output / context) from a chunk of source text. Eliminates the cold-start when building a new suite. | Yes (judge) |
| `eval_ingest_trace` | Convert a JSON agent trace (LangGraph / OpenAI Agents / manual) into an EvalCase payload. Use to score trajectories your agent just executed. | No |

## Example session

```
User: I just shipped a RAG endpoint. Can you check it for hallucinations?

Claude: I'll use multivon to evaluate it.
        [calls eval_discover to see what's available]
        [calls eval_faithfulness with your input/context/output]

→ score: 0.667 (passed: False), threshold: 0.9
  reason: 2/3 claims grounded
    ✓ "annual renewal" — supported by context
    ✓ "30-day notice" — supported by context
    ✗ "automatic upgrade" — NOT in context

Claude: Your RAG hallucinated the "automatic upgrade" detail. The context
        doesn't mention upgrades. I'd add a Hallucination evaluator to your CI
        gate, threshold ≥0.85, and re-prompt with explicit "only use facts
        from context" instructions.
```

## Release decisions from saved evidence

Use `eval_acceptance_report(report_json_path, policy_json_path)` to apply the
same `multivon.policy/v1` contract used by the CLI and CI action. It makes no
model calls. The result includes `decision` (`accept`, `reject`, `indeterminate`),
`exit_code`, `policy_digest`, `measurements` and `findings`. Malformed files or
policies produce an MCP tool error, not an accepted result. See the
[acceptance policy guide](https://docs.multivon.ai/guides/acceptance-policies).

Comparison is diagnostic: `eval_compare_runs` retains `identity_verified` and
`identity_issues`. Missing or incompatible evidence suppresses the paired
p-value. A large p-value does not prove equivalence, and an unchanged failing
application still needs an absolute release contract.

Evaluator responses include `status`, `measured` and original `metadata`.
For skipped or errored measurements, `score` and `passed` are **null**. Do not
coerce them into an ordinary pass or fail. Missing `agent_trace` means unknown;
provide `agent_trace: []` only for an observed empty trajectory. Trace ingestion
also requires an explicit `steps` list.

In single-call accuracy mode, supplied expected arguments must match exactly:
extra keys, absent null-valued keys and changed JSON types fail. Omitting
`expected_arguments` explicitly selects tool-name-only comparison. Trace mode
checks tool names/order, not argument correctness or external side effects.

These response semantics change in 0.4.0. Update clients that assume `passed`
and `score` are always measured values. The stdio integration test exercises
real initialization, discovery, acceptance, missing evidence and tool errors.

## Why these 23 tools (not all 44)

`eval_discover` returns the full 44-evaluator catalog, so the agent can always introspect everything. The 23 tools we expose directly are the ones agents actually call mid-edit:

- RAG generation checks (faithfulness, hallucination, relevance, answer_accuracy)
- RAG retrieval checks (context_precision, context_recall)
- Safety / fairness guardrails (toxicity, bias)
- Compliance (pii_detection, schema_compliance) — local-only, no API egress
- Flexible scoring (g_eval, custom_rubric) for user-defined rubrics
- Multimodal (vqa_faithfulness, document_grounding) for vision agents
- Agent traces (tool_call_accuracy)
- Document AI (`pdfhell_run`, `pdfhell_make`) — for any RAG-on-PDFs flow
- Audit pack — when procurement is involved
- Discover — meta-capability for planning
- Agent workflows (compare_runs, generate_cases, ingest_trace) — the loop that turns one-shot scoring into iterative improvement

The three new 0.3.0 tools exist because evals pay off as a loop: generate a starting suite from your own docs (`eval_generate_cases`), run your agent over it, score the trace (`eval_ingest_trace` → `eval_*`), make a fix, then verify the fix improved things vs. the baseline (`eval_compare_runs`). Agents need that whole loop callable from within a conversation, or they fall back to ad-hoc judgment.

Exposing all 44 evaluators as MCP tools would bloat the agent's context window and overwhelm tool-selection. If you need an evaluator that's not directly exposed, the agent can still use `multivon-eval` as a library — `eval_discover` returns the import paths.

## Evaluate saved execution evidence

The multivon-eval development checkout includes an
[OTLP evidence bridge](https://docs.multivon.ai/guides/otel-evidence).
Its example instruments an actual MCP stdio call with the official OpenTelemetry
SDK, grades the retained trace, then passes the saved report and policy to
`eval_acceptance_report` in published multivon-mcp 0.4.0. The complete synthetic
fixture is accepted; missing capture-completeness evidence is indeterminate.
No model API calls are required.

This is an opt-in library integration, not automatic server instrumentation or a
new MCP tool. The bridge is not included in multivon-eval 0.18.0; see its guide
for the checkout installation, native capture requirements and limitations.

The development [Gymnasium outcome bridge](https://docs.multivon.ai/guides/environment-outcomes)
also produces reports this server can evaluate. Three actual stdio calls verified
accept, reject and indeterminate decisions from saved SQLite outcome evidence.
The adapter checks persisted state and forbidden changes; it is a development
preview, not part of the server's pinned multivon-eval 0.18.0 dependency.

The development [media evidence bridge](https://docs.multivon.ai/guides/media-evidence)
binds native Inspect media to exact content and explicit verdict references.
Published MCP 0.4.0 reproduced a rejection from its saved six-case document
report over stdio. This tool reads the report and policy; it does not fetch or
decode media itself. The new media APIs require the core development checkout
and are not added to this server's pinned dependency by this example.

## Dependencies

Tested runtime bounds (from `pyproject.toml`):

- `mcp[cli] >= 1.29, < 2` — official MCP Python SDK and Inspector. MCP 2.0 has a different server API and is intentionally excluded until this server migrates.
- `multivon-eval >= 0.18.0, < 0.19` — the 44-evaluator engine, current report schema, and reasoning-judge fix.
- `pdfhell >= 0.6.2, < 0.7` — the 17-family mini-v4 registry, corrected trap renderings, and current audit-pack schema.

These bounds ship in 0.4.0. Upgrade with `pip install -U multivon-mcp`.

All Apache 2.0.

## MCP server vs Claude Code skills vs eval-action — which one do I use?

`multivon-eval` ships three agent-facing surfaces. They overlap on what
they call (the same evaluator catalog) but differ on where the agent
lives.

| Surface | Where the agent runs | Best for |
|---|---|---|
| **multivon-mcp** (this repo) | Any MCP-compatible client — Claude Desktop, Cursor, Cline, OpenCode, Claude Code | Mid-edit scoring inside an IDE or chat app. Agent calls `eval_faithfulness` / `eval_hallucination` / etc. directly as tools. |
| **Claude Code skills** — `eval-bootstrap`, `eval-audit`, `eval-explain` (bundled in `multivon-eval >= 0.9.8`; install with `multivon-eval install-skills`) | Claude Code only | Workflow-shaped tasks: scaffold an eval suite from a project description, pre-PR regression checks against a baseline, explaining why a particular evaluator was picked. The skills know how to call `multivon-eval bootstrap` / use `compare_reports` / etc. so the agent doesn't have to figure it out from docs. |
| **[eval-action](https://github.com/multivon-ai/eval-action)** | GitHub CI | Gate every PR on eval regressions automatically. Posts the Wilson-CI + McNemar verdict as a PR comment. |

If you're building an LLM product and want the agent in your editor to
score a RAG output without copy-pasting Python, use multivon-mcp.
If you live in Claude Code and want the bootstrap → audit → explain
loop wired up as native commands, use the bundled skills. For PR-time
gating, use the GitHub Action. Most projects end up using more than
one.

## The Multivon ecosystem

Four public packages plus one closed early-access product, built around the same evaluation engine:

| Repo | What it is |
|---|---|
| [multivon-eval](https://github.com/multivon-ai/multivon-eval) | Python SDK — 44 evaluators + `bootstrap` CLI + `multivon_eval.auto`. The engine multivon-mcp wraps. |
| [pdfhell](https://github.com/multivon-ai/pdfhell) | Adversarial PDFs that break AI document readers — exposed here as `pdfhell_run` + `pdfhell_make` tools |
| **multivon-mcp** (you are here) | MCP server — 23 tools from multivon-eval + pdfhell |
| [eval-action](https://github.com/multivon-ai/eval-action) | GitHub Action — runs the same evals on every PR |
| multivon-guard *(early access)* | Local proxy that catches LLM coding agents leaking secrets / PII |

## License

Apache 2.0.

## Citing

```bibtex
@software{multivon_mcp,
  title  = {multivon-mcp: MCP server exposing multivon-eval + pdfhell as agent-callable tools},
  author = {Multivon},
  year   = {2026},
  url    = {https://github.com/multivon-ai/multivon-mcp},
}
```

The experimental [vector world-model bridge](https://docs.multivon.ai/guides/world-models)
produces the same saved report format. Actual stdio checks using published MCP
0.4.0 and core 0.18.0 reproduced accept/reject/indeterminate for measured versus
missing CartPole forecast checks. The server ran outside the development
checkout so the installed core wheel was used. These post-hoc checks establish
transport and coverage behavior, not world-model quality or a release decision.
Training, simulation and prediction remain in the upstream library workflow;
this server adds no world-model runner or heavy learning dependencies.