Skip to main content
Glama
multivon-mcp

multivon-mcp

Official
by multivon-ai

multivon-mcp

PyPI Python License Downloads

Docs · Website · PyPI · multivon-eval (engine) · Changelog

These 23 tools cover what an autonomous eval agent needs to do its job: discover its own capabilities (eval_discover), normalize traces from supported sources (eval_ingest_trace), and run evaluators against them, with calibration evidence specific to the tested task. We put the framework behind an MCP boundary because eval belongs in the agent's working loop, not behind a separate dashboard.

An MCP server that gives AI coding agents direct access to evaluation tools. Drop into Claude Desktop, Claude Code, Cursor, Cline, or any Model Context Protocol–compatible agent.

When the agent is helping you build an LLM product, it can:

  • Score a RAG output for hallucination without you writing the scaffolding

  • Generate an adversarial PDF on demand to test your document AI

  • Run the full pdfhell mini-suite against a model and analyse the results

  • Produce a self-verifying audit pack with a SHA-256 file manifest

  • Discover the full evaluation capability catalog as JSON

No copy-paste, and no asking the agent to figure out the SDK calls from python -c "..." one-liners.

Current release: 0.4.0. Tested with MCP Python SDK 1.29.x, multivon-eval 0.18.0, and pdfhell 0.6.2. See the changelog.

Install

pip install "multivon-mcp==0.4.0"

The next release carries this compatibility bound itself. Installation pulls multivon-eval, pdfhell, and the MCP SDK. The provider SDKs (anthropic, openai, google-genai) come along too — bring your own API key in env.

Related MCP server: agent-comm

Configure your agent

Claude Code

claude mcp add --transport stdio --scope user multivon -- multivon-mcp
claude mcp get multivon

Set provider keys in your shell or secure environment before starting Claude Code. To share the server configuration with a project instead, use --scope project; Claude Code writes .mcp.json and supports environment-variable expansion there. It does not read claude_desktop_config.json.

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "multivon": {
      "command": "multivon-mcp",
      "env": {
        "ANTHROPIC_API_KEY": "sk-ant-...",
        "OPENAI_API_KEY": "sk-proj-...",
        "GOOGLE_API_KEY": "AIza..."
      }
    }
  }
}

Restart Claude. The 23 tools become available; ask Claude "use multivon to evaluate this RAG output" and it figures out which tool to call.

Cursor

.cursor/mcp.json or via Settings → MCP:

{ "mcpServers": { "multivon": { "command": "multivon-mcp" } } }

Cline / OpenCode / any MCP-compatible agent

Same shape — point at the multivon-mcp console script.

Local dev / debugging

From a clone of this repo:

mcp dev multivon_mcp/server.py

From a pip install (the file lives in site-packages, so resolve it):

mcp dev "$(python -c 'import multivon_mcp.server as s; print(s.__file__)')"

Opens the MCP Inspector UI in your browser. You can call any tool by name, see the JSON schemas, and watch the requests/responses.

The 23 tools

Discovery & document AI

Tool

What it does

API key

eval_discover

Full machine-readable capability catalog (evaluators, traps, suites, calibration data, versions). Call first.

No

pdfhell_make

Generate one adversarial PDF + its answer key.

No

pdfhell_run

Run the pdfhell adversarial-PDF benchmark against a vision model. Returns pass rate, per-trap CIs, suite hash.

Yes (vision)

eval_audit_pack

Build a procurement-ready ZIP with a SHA-256 file manifest from a pdfhell run.

No

RAG generation & retrieval

Tool

What it does

API key

eval_faithfulness

QAG-graded faithfulness — is a RAG output grounded in the retrieved context?

Yes

eval_hallucination

QAG-graded hallucination — does the output contain content NOT in context?

Yes

eval_relevance

QAG-graded answer-vs-question relevance.

Yes

eval_answer_accuracy

QAG-graded semantic equivalence vs ground truth.

Yes

eval_context_precision

RAG retrieval quality — are the retrieved chunks on-topic?

Yes

eval_context_recall

RAG retrieval completeness — does context contain enough info to answer?

Yes

Safety, compliance, fairness

Tool

What it does

API key

eval_toxicity

QAG-graded toxicity / harmful-content detection.

Yes

eval_bias

QAG-graded bias across gender, race, politics, age, socioeconomic axes.

Yes

eval_pii_detection

Local-only regex scan for PII (GDPR / CCPA / PIPEDA / HIPAA / DPDP packs).

No

eval_schema_compliance

Validate an LLM output against a JSON Schema.

No

Agent & multimodal

Tool

What it does

API key

eval_tool_call_accuracy

Deterministic agent tool-call correctness. No LLM.

No

eval_vqa_faithfulness

Image-grounded visual-QA faithfulness.

Yes (vision)

eval_document_grounding

Multi-page document-grounded faithfulness for document-AI agents.

Yes (vision)

Agent traces. eval_tool_call_accuracy and the other agent-trace evaluators in multivon-eval (ToolArgumentAccuracy, ToolCallNecessity, TrajectoryEfficiency, AgentMemoryEval, PlanQuality, TaskCompletion, StepFaithfulness) take an agent_trace=[AgentStep(...)] plus expected_tool_calls=[...] on the case. Three-shape semantics matter: expected_tool_calls=None skips, [] asserts "no tools called", and [...] checks for the named calls. On repository main (shipping in the next release), the MCP tool supports the same trace mode: normalize trace JSON with eval_ingest_trace, then pass its agent_trace plus expected_tool_calls to eval_tool_call_accuracy. Set require_order=true when sequence matters or penalize_unexpected=true for a strict allow-list. See the multivon-eval agent integrations for the source-of-truth tracer code.

Flexible scoring

Tool

What it does

API key

eval_g_eval

G-Eval holistic 0.0-1.0 scoring against a plain-English criterion.

Yes

eval_custom_rubric

Score against your own list of yes/no quality checks.

Yes

Agent workflows (new in 0.3.0)

Tool

What it does

API key

eval_acceptance_report

Apply required-check, coverage and slice policy to a saved report; returns accept/reject/indeterminate.

No

eval_compare_runs

Diff two eval report JSONs — pass-rate delta, per-case regressions/improvements, McNemar p-value. Includes identity warnings; inconclusive comparisons cannot confirm improvement.

No

eval_generate_cases

Generate N eval cases (input / expected_output / context) from a chunk of source text. Eliminates the cold-start when building a new suite.

Yes (judge)

eval_ingest_trace

Convert a JSON agent trace (LangGraph / OpenAI Agents / manual) into an EvalCase payload. Use to score trajectories your agent just executed.

No

Example session

User: I just shipped a RAG endpoint. Can you check it for hallucinations?

Claude: I'll use multivon to evaluate it.
        [calls eval_discover to see what's available]
        [calls eval_faithfulness with your input/context/output]

→ score: 0.667 (passed: False), threshold: 0.9
  reason: 2/3 claims grounded
    ✓ "annual renewal" — supported by context
    ✓ "30-day notice" — supported by context
    ✗ "automatic upgrade" — NOT in context

Claude: Your RAG hallucinated the "automatic upgrade" detail. The context
        doesn't mention upgrades. I'd add a Hallucination evaluator to your CI
        gate, threshold ≥0.85, and re-prompt with explicit "only use facts
        from context" instructions.

Release decisions from saved evidence

Use eval_acceptance_report(report_json_path, policy_json_path) to apply the same multivon.policy/v1 contract used by the CLI and CI action. It makes no model calls. The result includes decision (accept, reject, indeterminate), exit_code, policy_digest, measurements and findings. Malformed files or policies produce an MCP tool error, not an accepted result. See the acceptance policy guide.

Comparison is diagnostic: eval_compare_runs retains identity_verified and identity_issues. Missing or incompatible evidence suppresses the paired p-value. A large p-value does not prove equivalence, and an unchanged failing application still needs an absolute release contract.

Evaluator responses include status, measured and original metadata. For skipped or errored measurements, score and passed are null. Do not coerce them into an ordinary pass or fail. Missing agent_trace means unknown; provide agent_trace: [] only for an observed empty trajectory. Trace ingestion also requires an explicit steps list.

In single-call accuracy mode, supplied expected arguments must match exactly: extra keys, absent null-valued keys and changed JSON types fail. Omitting expected_arguments explicitly selects tool-name-only comparison. Trace mode checks tool names/order, not argument correctness or external side effects.

These response semantics change in 0.4.0. Update clients that assume passed and score are always measured values. The stdio integration test exercises real initialization, discovery, acceptance, missing evidence and tool errors.

Why these 23 tools (not all 44)

eval_discover returns the full 44-evaluator catalog, so the agent can always introspect everything. The 23 tools we expose directly are the ones agents actually call mid-edit:

  • RAG generation checks (faithfulness, hallucination, relevance, answer_accuracy)

  • RAG retrieval checks (context_precision, context_recall)

  • Safety / fairness guardrails (toxicity, bias)

  • Compliance (pii_detection, schema_compliance) — local-only, no API egress

  • Flexible scoring (g_eval, custom_rubric) for user-defined rubrics

  • Multimodal (vqa_faithfulness, document_grounding) for vision agents

  • Agent traces (tool_call_accuracy)

  • Document AI (pdfhell_run, pdfhell_make) — for any RAG-on-PDFs flow

  • Audit pack — when procurement is involved

  • Discover — meta-capability for planning

  • Agent workflows (compare_runs, generate_cases, ingest_trace) — the loop that turns one-shot scoring into iterative improvement

The three new 0.3.0 tools exist because evals pay off as a loop: generate a starting suite from your own docs (eval_generate_cases), run your agent over it, score the trace (eval_ingest_trace → eval_*), make a fix, then verify the fix improved things vs. the baseline (eval_compare_runs). Agents need that whole loop callable from within a conversation, or they fall back to ad-hoc judgment.

Exposing all 44 evaluators as MCP tools would bloat the agent's context window and overwhelm tool-selection. If you need an evaluator that's not directly exposed, the agent can still use multivon-eval as a library — eval_discover returns the import paths.

Evaluate saved execution evidence

The multivon-eval development checkout includes an OTLP evidence bridge. Its example instruments an actual MCP stdio call with the official OpenTelemetry SDK, grades the retained trace, then passes the saved report and policy to eval_acceptance_report in published multivon-mcp 0.4.0. The complete synthetic fixture is accepted; missing capture-completeness evidence is indeterminate. No model API calls are required.

This is an opt-in library integration, not automatic server instrumentation or a new MCP tool. The bridge is not included in multivon-eval 0.18.0; see its guide for the checkout installation, native capture requirements and limitations.

The development Gymnasium outcome bridge also produces reports this server can evaluate. Three actual stdio calls verified accept, reject and indeterminate decisions from saved SQLite outcome evidence. The adapter checks persisted state and forbidden changes; it is a development preview, not part of the server's pinned multivon-eval 0.18.0 dependency.

The development media evidence bridge binds native Inspect media to exact content and explicit verdict references. Published MCP 0.4.0 reproduced a rejection from its saved six-case document report over stdio. This tool reads the report and policy; it does not fetch or decode media itself. The new media APIs require the core development checkout and are not added to this server's pinned dependency by this example.

Dependencies

Tested runtime bounds (from pyproject.toml):

  • mcp[cli] >= 1.29, < 2 — official MCP Python SDK and Inspector. MCP 2.0 has a different server API and is intentionally excluded until this server migrates.

  • multivon-eval >= 0.18.0, < 0.19 — the 44-evaluator engine, current report schema, and reasoning-judge fix.

  • pdfhell >= 0.6.2, < 0.7 — the 17-family mini-v4 registry, corrected trap renderings, and current audit-pack schema.

These bounds ship in 0.4.0. Upgrade with pip install -U multivon-mcp.

All Apache 2.0.

MCP server vs Claude Code skills vs eval-action — which one do I use?

multivon-eval ships three agent-facing surfaces. They overlap on what they call (the same evaluator catalog) but differ on where the agent lives.

Surface

Where the agent runs

Best for

multivon-mcp (this repo)

Any MCP-compatible client — Claude Desktop, Cursor, Cline, OpenCode, Claude Code

Mid-edit scoring inside an IDE or chat app. Agent calls eval_faithfulness / eval_hallucination / etc. directly as tools.

Claude Code skills — eval-bootstrap, eval-audit, eval-explain (bundled in multivon-eval >= 0.9.8; install with multivon-eval install-skills)

Claude Code only

Workflow-shaped tasks: scaffold an eval suite from a project description, pre-PR regression checks against a baseline, explaining why a particular evaluator was picked. The skills know how to call multivon-eval bootstrap / use compare_reports / etc. so the agent doesn't have to figure it out from docs.

eval-action

GitHub CI

Gate every PR on eval regressions automatically. Posts the Wilson-CI + McNemar verdict as a PR comment.

If you're building an LLM product and want the agent in your editor to score a RAG output without copy-pasting Python, use multivon-mcp. If you live in Claude Code and want the bootstrap → audit → explain loop wired up as native commands, use the bundled skills. For PR-time gating, use the GitHub Action. Most projects end up using more than one.

The Multivon ecosystem

Four public packages plus one closed early-access product, built around the same evaluation engine:

Repo

What it is

multivon-eval

Python SDK — 44 evaluators + bootstrap CLI + multivon_eval.auto. The engine multivon-mcp wraps.

pdfhell

Adversarial PDFs that break AI document readers — exposed here as pdfhell_run + pdfhell_make tools

multivon-mcp (you are here)

MCP server — 23 tools from multivon-eval + pdfhell

eval-action

GitHub Action — runs the same evals on every PR

multivon-guard (early access)

Local proxy that catches LLM coding agents leaking secrets / PII

License

Apache 2.0.

Citing

@software{multivon_mcp,
  title  = {multivon-mcp: MCP server exposing multivon-eval + pdfhell as agent-callable tools},
  author = {Multivon},
  year   = {2026},
  url    = {https://github.com/multivon-ai/multivon-mcp},
}

The experimental vector world-model bridge produces the same saved report format. Actual stdio checks using published MCP 0.4.0 and core 0.18.0 reproduced accept/reject/indeterminate for measured versus missing CartPole forecast checks. The server ran outside the development checkout so the installed core wheel was used. These post-hoc checks establish transport and coverage behavior, not world-model quality or a release decision. Training, simulation and prediction remain in the upstream library workflow; this server adds no world-model runner or heavy learning dependencies.

Available Tools

23 tools
eval_acceptance_reportA

Apply a versioned acceptance policy to saved evidence without model calls.

Read a full EvalReport JSON and a multivon.policy/v1 JSON policy. Return accept, reject, or indeterminate, the policy digest, measured coverage, per-slice results and findings. Missing required checks or trials cannot pass. A known quality failure can reject even with incomplete evidence. The returned exit_code is the policy result, not the MCP process status. Input/configuration failures are MCP tool errors, never accepted reports.

Inputs: policy_json_path (required), report_json_path (required). Returns: object

eval_answer_accuracyA

Evaluate whether an answer is semantically equivalent to the ground truth.

QAG-graded — generates yes/no questions about whether the actual answer matches the meaning of the expected answer. Useful when string match is too strict (e.g. paraphrased correct answers).

Args: expected_answer: Ground-truth answer. actual_answer: The LLM's answer. judge_model: Provider:model for the QAG judge.

Returns: {"score": 0.0-1.0, "passed": bool, "reason": str}.

Inputs: actual_answer (required), expected_answer (required), judge_model. Returns: object

eval_audit_packA

Build a self-verifying audit ZIP from a pdfhell run.

Combines the run JSON, the case PDFs + answer keys, JUnit XML, and a SHA-256 manifest into one downloadable ZIP. Suitable for attaching to a procurement diligence appendix. Anchor the manifest hash outside the ZIP when you need evidence against deliberate rewriting; the ZIP alone is self-verifying, not tamper-proof.

Args: run_json_path: Path to a pdfhell run JSON (from pdfhell run --out). cases_dir: Directory containing the case PDFs + answer keys that were evaluated. Same dir the run used. output_zip_path: Where to write the audit ZIP.

Returns: {"path": "/abs/path/to.zip", "size_bytes": N, "manifest": {...}}. The manifest dict mirrors the one inside the ZIP — useful for an agent that wants to verify the contents without opening the ZIP itself.

Inputs: cases_dir (required), run_json_path (required), output_zip_path (required). Returns: object

eval_biasA

Detect gender, racial, political, age, or socioeconomic bias.

QAG-graded — the judge answers five yes/no questions about whether the response treats demographic groups equitably and stays politically neutral. Score 1.0 = no detectable bias, 0.0 = significant bias.

Use this for fairness audits on generated content, summarisation of user-submitted text, or hiring/lending-adjacent LLM pipelines.

Args: input: The prompt the LLM was responding to. Bias evaluators use this to contextualise whether disparate treatment is justified (e.g. a medical question may legitimately discuss group-specific risk factors). output: The LLM-generated response. judge_model: Provider:model for the QAG judge.

Returns: {"score": 0.0-1.0, "passed": bool, "reason": str, "threshold": float, "evaluator": "bias"}.

Inputs: input (required), output (required), judge_model. Returns: object

eval_compare_runsA

Compare two multivon-eval report JSONs and return a structured diff.

Loads both reports from disk (the JSON produced by EvalReport.to_json()), pairs cases by stable ID and case digest, and returns pass-rate / average-score deltas plus the per-case regressions and improvements lists. Includes a McNemar p-value only when pairing evidence supports it. A large p-value does not establish equivalence. Legacy reports remain diagnostic.

Use this when you've made a prompt / retrieval / model change and want to know if the new run actually improved over the baseline — not just on aggregate, but case-by-case.

Args: baseline_json_path: Filesystem path to the baseline report JSON (e.g. "runs/baseline.json"). new_json_path: Filesystem path to the new / proposal report JSON to compare against the baseline.

Returns: A dict with: - pass_rate_delta: float, new - baseline pass rate - avg_score_delta: float, new - baseline average score - regressions: list of dicts with input, baseline_status, proposal_status, baseline_score, proposal_score - improvements: same shape as regressions - mcnemar_p_value: float or null — paired-test p-value - baseline / proposal: summary blocks with name, pass_rate, avg_score, errors, flaky - paired_count / added_count / removed_count: pairing stats so the caller can see how many cases lined up vs. drifted between runs

Inputs: new_json_path (required), baseline_json_path (required). Returns: object

eval_context_precisionA

Measure whether retrieved RAG context chunks are relevant to the question.

High precision = the retriever returned mostly on-topic chunks; low noise. The judge asks "is this chunk relevant?" for each chunk (up to 8) and scores precision = fraction marked relevant.

Use this to diagnose retriever quality: if precision is low, your embedding model, chunk size, or reranker is returning noise.

Args: input: The user's question. context: Either a list of retrieved chunks, or a single string with the full retrieved context (will be evaluated as one chunk). judge_model: Provider:model for the QAG judge.

Returns: {"score": 0.0-1.0, "passed": bool, "reason": str, "threshold": float, "evaluator": "context_precision"}.

Inputs: input (required), context (required), judge_model. Returns: object

eval_context_recallA

Measure whether retrieved context contains enough information to answer.

High recall = the retriever found the information needed to derive the expected answer. The judge asks whether the expected answer could plausibly be reconstructed from the retrieved context alone.

Use this when you have a labelled QA dataset and want to diagnose whether failures are retriever misses vs. generator errors.

Args: input: The user's question. context: The retrieved context chunks (list or single string). expected_answer: The ground-truth answer the context should support. judge_model: Provider:model for the QAG judge.

Returns: {"score": 0.0-1.0, "passed": bool, "reason": str, "threshold": float, "evaluator": "context_recall"}.

Inputs: input (required), context (required), expected_answer (required), judge_model. Returns: object

eval_custom_rubricA

Score an output against your own list of yes/no quality checks.

Each criterion is a [question, expect_yes] pair. The judge answers each question with yes/no; the score is the fraction answered as expected. Best for compliance-style rubrics where each aspect should be auditable separately.

Args: input: The prompt the LLM was responding to. output: The LLM-generated response. criteria: A list of [question_str, expect_yes_bool] pairs. Example: [["Does it cite a source?", true], ["Does it speculate beyond the source?", false]]. name: Optional label for the rubric (appears in the result dict's evaluator field). context: Optional context string for the judge to consider (e.g. retrieved RAG context, source document). judge_model: Provider:model for the QAG judge.

Returns: {"score": 0.0-1.0, "passed": bool, "reason": str, "threshold": float, "evaluator": <name>}.

Inputs: input (required), output (required), criteria (required), name, +2 more. Returns: object

eval_discoverA

Return the full machine-readable capability catalog.

Useful as a first call at session start — an agent can plan its evaluation strategy against the actual available evaluators rather than guessing or hallucinating tool names.

Returns: A dict with six top-level keys:

- ``evaluators``: every available multivon-eval evaluator,
  with its category, import path, evaluator ID, and summary.
- ``traps``: every pdfhell trap family, the failure mode each
  elicits, and the expected_failure_mode metadata.
- ``suites``: every named pdfhell suite, the (trap_family,
  seed_count) breakdown, and the suite_hash for the canonical
  version.
- ``calibration``: shipped per-(evaluator, judge) threshold rows.
- ``version``: installed multivon-mcp, multivon-eval, and pdfhell
  versions.
- ``server``: the stable server identifier.

Inputs: no named parameters. Returns: object

eval_document_groundingA

Check whether an answer about a multi-page document is grounded.

Document-page-grounded faithfulness for multi-page document agents (contracts, invoices, scientific PDFs, medical records). The vision judge answers three yes/no questions per document: is every claim supported, no inventions, exceptions handled.

Provide one image per page. Use exactly one of:

  • images: list of paths, http(s) URLs, or data URIs.

  • images_base64: list of raw base64 strings; pair with mime_type.

Args: input: The question or prompt the LLM was answering about the document. output: The LLM-generated answer to verify against the pages. images: List of page image sources (paths/URLs/data URIs). images_base64: Alternative — list of raw base64 strings. mime_type: Mime type when using images_base64. Default "image/png". judge_model: Provider:model for the vision judge. Must be vision-capable. Default "google:gemini-2.5-flash".

Returns: {"score": 0.0-1.0, "passed": bool, "reason": str, "threshold": float, "evaluator": "document_grounding"}.

Inputs: input (required), output (required), images, mime_type, +2 more. Returns: object

eval_faithfulnessA

Evaluate whether an LLM output is grounded in the retrieved context.

Uses multivon-eval's QAG-graded Faithfulness evaluator. Extracts factual claims from the output and verifies each one against the context. Score is the fraction of claims supported.

Use this when a RAG pipeline returned an answer and you want to check the LLM didn't invent facts not present in retrieved documents.

Args: input: The user's question. context: The retrieved context the LLM was given. output: The LLM's answer being evaluated. judge_model: Provider:model for the QAG judge. Default "anthropic:claude-haiku-4-5" (cheap + calibrated).

Returns: {"score": 0.0-1.0, "passed": bool, "reason": str, "threshold": float}.

Inputs: input (required), output (required), context (required), judge_model. Returns: object

eval_generate_casesA

Generate synthetic eval cases from a source text.

Calls multivon-eval's synthetic generator to produce n eval cases from raw text (docs, FAQ, knowledge base). Each case has an input (question), expected_output (ground-truth answer), and context (the source excerpt the answer was grounded in). Eliminates the cold-start problem when building a new eval suite from scratch.

Requires a provider API key in env so the underlying judge can propose question/answer pairs.

Args: from_text: Source text to generate cases from (e.g. FAQ, docs chunk, knowledge base article). n: Number of cases to generate. Default 10. task: One of "qa" (question/answer pairs — default), "summarization" (text + expected summary), or "hallucination" (faithful answer + expected_output = "faithful" for hallucination benchmarks). judge_model: Provider:model string used to generate the cases. The generator calls this judge under the hood; it does NOT need to match the judge you eventually use to evaluate the cases. Default "anthropic:claude-haiku-4-5".

Returns: A list of dicts {"input", "expected_output", "context", "metadata"} ready to feed into EvalCase(**d) or to persist as a JSONL eval dataset.

Inputs: from_text (required), n, task, judge_model. Returns: result

eval_g_evalA

G-Eval style holistic scoring against a plain-English criterion.

The judge reads the criterion and the output, then returns a numeric score from 0.0 to 1.0 plus a short reason. To reduce single-sample variance the prompt is run twice by default and the scores averaged (position/framing bias mitigation per the original G-Eval paper).

Best for fuzzy or holistic qualities: creativity, tone, style, helpfulness, conciseness. For criteria with multiple discrete aspects, prefer eval_custom_rubric.

Args: input: The prompt the LLM was responding to. output: The LLM-generated response to score. criteria: A plain-English description of what to score on, e.g. "Is the response concise, polite, and free of jargon?". name: Optional label for the evaluator instance (appears in the result dict's evaluator field). runs: How many independent judgements to average. Default 2. judge_model: Provider:model for the scoring judge.

Returns: {"score": 0.0-1.0, "passed": bool, "reason": str, "threshold": float, "evaluator": <name>}.

Inputs: input (required), output (required), criteria (required), name, +2 more. Returns: object

eval_hallucinationB

Detect fabricated information not present in the context.

Score 1.0 = no hallucination. Score 0.0 = significant hallucination.

Args: output: The LLM output to check. context: The ground-truth context the output should be grounded in. judge_model: Provider:model for the QAG judge.

Returns: {"score": 0.0-1.0, "passed": bool, "reason": str, "threshold": float}.

Inputs: output (required), context (required), judge_model. Returns: object

eval_ingest_traceA

Convert a JSON agent trace into a JSON-friendly EvalCase payload.

Parses a serialised agent trajectory and returns the :class:EvalCase shape the rest of the eval pipeline (and the other eval_* MCP tools) expect. Use this when your agent has just finished a trajectory at runtime and you want to score that trajectory immediately — no need to re-run anything.

Supports three frameworks:

  • "langgraph" (default): canonical universal step list

  • "openai_agents": canonical OR {"new_items": [...]} from a RunResult you serialised

  • "manual": canonical step list

Args: trace_json: The trace as a JSON-friendly dict. Must include input; steps (or new_items for openai_agents) is strongly recommended. framework: One of "langgraph", "openai_agents", "manual". Defaults to "langgraph".

Returns: A dict with input, expected_output, context, expected_tool_calls, agent_trace (list of step dicts), and metadata — ready to feed back into other eval_* MCP tools or to persist as part of an eval dataset.

Inputs: trace_json (required), framework. Returns: object

eval_pii_detectionA

Detect personally-identifiable information (PII) in an LLM output.

Local-first: zero API calls. Uses a regex pattern library covering emails, phone numbers, SSNs, credit cards, IBANs, IPs, addresses, and jurisdiction-specific identifiers (HIPAA MRNs, EU VAT, California bank accounts, etc).

Score 1.0 = no PII detected. Score 0.0 = PII found (the reason field lists which types matched and example substrings).

Args: output: The LLM-generated text to scan. jurisdiction: Which extra pattern set to include. One of "all" (default), "gdpr", "ccpa", "pipeda", "hipaa", or "dpdp" (India). custom_patterns: Optional dict of {name: regex} to add to the default library (e.g. {"employee_id": r"EMP-\d{6}"}). redact: If True, replace matched substrings with [REDACTED-TYPE] markers in the reason field.

Returns: {"score": 0.0 or 1.0, "passed": bool, "reason": str, "threshold": float, "evaluator": "pii_detection"}.

Inputs: output (required), redact, jurisdiction, custom_patterns. Returns: object

eval_relevanceA

Check whether an LLM output actually addresses the user's question.

QAG-graded — generates yes/no questions about whether the output answers the input, stays on topic, contains relevant content.

Args: input: The user's question. output: The LLM's response. judge_model: Provider:model for the QAG judge.

Returns: {"score": 0.0-1.0, "passed": bool, "reason": str, "threshold": float}.

Inputs: input (required), output (required), judge_model. Returns: object

eval_schema_complianceA

Validate that an LLM output conforms to a JSON Schema.

Wraps multivon-eval's SchemaEvaluator. Parses the LLM output as JSON (tolerantly strips markdown code fences), then validates the parsed structure against the provided JSON Schema dict. Reports per-field validation errors — not just "valid/invalid".

For Pydantic-model validation or more advanced setups (custom validators, recursive schemas), use the multivon-eval SDK directly.

Args: output: The LLM-generated text expected to contain JSON. schema: A JSON Schema dict (Draft 7). Example: {"type": "object", "required": ["title", "score"], "properties": {"title": {"type": "string"}, "score": {"type": "number"}}}. strict: If True, additional fields not in the schema are treated as failures.

Returns: {"score": 0.0-1.0, "passed": bool, "reason": str, "threshold": float, "evaluator": "schema_compliance"}.

Inputs: output (required), schema (required), strict. Returns: object

eval_tool_call_accuracyA

Evaluate whether an agent called the expected tool or tool sequence.

Pure deterministic — no LLM judge needed. Two compatible modes:

  • Single-call mode compares expected_tool / actual_tool and optional argument dictionaries exactly.

  • Trace mode consumes expected_tool_calls plus the canonical agent_trace returned by eval_ingest_trace. It can require order and optionally penalize unexpected calls.

Args: expected_tool: Single tool name the agent should have called. actual_tool: Single tool name the agent actually called. expected_arguments: Expected arguments for single-call mode. actual_arguments: Actual arguments for single-call mode. expected_tool_calls: Expected names for trace mode. An empty list explicitly asserts that the agent should call no tools. agent_trace: Canonical step dictionaries returned by eval_ingest_trace. require_order: In trace mode, require expected names in order. penalize_unexpected: In trace mode, lower the score for calls not present in expected_tool_calls.

Returns: {"score": float, "passed": bool, "reason": str, "evaluator": "tool_call_accuracy"}, or an error dict when the arguments do not form either mode.

Inputs: actual_tool, agent_trace, expected_tool, require_order, +4 more. Returns: object

eval_toxicityA

Detect harmful, offensive, or inappropriate content in an LLM output.

QAG-graded — the judge answers four yes/no questions about whether the output is free of hate speech, threats, harmful instructions, and disrespectful tone. Score is the fraction of questions that pass; 1.0 = not toxic, 0.0 = toxic.

Use this for guardrails on generated content, chatbot turns, or any user-facing LLM output where harmful content is a risk.

Args: output: The LLM-generated text to evaluate. judge_model: Provider:model for the QAG judge, e.g. "anthropic:claude-haiku-4-5" (default), "openai:gpt-4o-mini", or "google:gemini-2.5-flash".

Returns: {"score": 0.0-1.0, "passed": bool, "reason": str, "threshold": float, "evaluator": "toxicity"}.

Inputs: output (required), judge_model. Returns: object

eval_vqa_faithfulnessA

Check whether an LLM answer about an image is grounded in what's visible.

Image-grounded faithfulness. The vision judge extracts up to 3 factual claims from the answer, then verifies each one against the image. Score = fraction of claims that are accurate.

Use this for visual QA, image captioning, chart/diagram reading, and any LLM output that purports to describe an image.

Image input — exactly one of:

  • image: a local path, http(s) URL, or full data URI.

  • image_base64: raw base64 (no data: prefix); pair with mime_type (default "image/png").

Args: input: The question or prompt the LLM was answering. output: The LLM-generated answer to verify against the image. image: Path / URL / data URI for the image. image_base64: Alternative — raw base64 image bytes. mime_type: Mime type when using image_base64. Default "image/png". Other common values: "image/jpeg", "image/webp". judge_model: Provider:model for the vision judge. Must be vision-capable. Default "google:gemini-2.5-flash" (cheap). Other vision-capable options: "openai:gpt-4o-mini" or "anthropic:claude-sonnet-4-6". Haiku 4.5 also supports image input; verify task quality on your own held-out cases.

Returns: {"score": 0.0-1.0, "passed": bool, "reason": str, "threshold": float, "evaluator": "vqa_faithfulness"}.

Inputs: input (required), output (required), image, mime_type, +2 more. Returns: object

pdfhell_makeA

Generate one adversarial PDF + its answer key.

Useful for an agent to inspect what a specific trap looks like before deciding to evaluate against it.

Args: trap: Trap family name. The full list of 17+ families is discoverable via eval_discover (which is also the source of truth — pdfhell adds families over time and hard-coding them here would go stale). Examples include "hidden_ocr_mismatch", "footnote_override", and the autoresearch-discovered families in mini-v3/v4. seed: Integer seed. Same seed → byte-identical PDF + identical answer key. return_pdf_bytes: If True, include the base64-encoded PDF bytes in the response. Default False — most agents want the question / expected answer, not the raw PDF.

Returns: A dict with the case JSON (id, trap_family, question, expected_answer, forbidden_answers, metadata) and optionally the base64-encoded PDF bytes under pdf_base64.

Inputs: seed (required), trap (required), return_pdf_bytes. Returns: object

pdfhell_runA

Run the pdfhell adversarial-PDF benchmark against a vision model.

Args: model: Provider:model spec, e.g. "anthropic:claude-sonnet-4-6", "openai:gpt-4o", "google:gemini-2.5-flash". suite: Any suite from eval_discover. Current suites: "smoke" (3 cases, ~10s), "mini" (30 cases, ~$0.01 on Flash), "mini-v2", "mini-v3", the flagship "mini-v4" (17 trap families, 510 cases), and "mini-v4-sample" (170 cases — cheap reproduction of mini-v4). Default "mini". workers: Parallel API requests. Default 4.

Returns: A dict with overall pass_rate, Wilson 95% CI, per-trap-family pass rates and CIs, and per-case details. Suite version + hash included so consumers can verify the run measured the expected cases.

Provider API keys come from environment variables (ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY) — not passed through this tool, never logged.

Inputs: model (required), suite, workers. Returns: object

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.9/5.0

    Scored across 23 tools

    Disambiguation3/5

    Most eval tools target distinct metrics, but several are easy to conflate: eval_faithfulness/eval_hallucination are inverse measures of the same construct, eval_vqa_faithfulness/eval_document_grounding both do vision-grounded checking, and eval_relevance/eval_answer_accuracy both score response quality. The detailed descriptions help, but the sheer number of similar score/pass/reason evaluators still creates real selection ambiguity.

    Naming Consistency4/5

    All tools use snake_case and nearly all share the eval_ prefix followed by a metric noun (eval_toxicity, eval_context_precision), with a few verb-style exceptions (eval_discover, eval_ingest_trace, eval_generate_cases). The two pdfhell_* tools form a coherent sub-namespace rather than a violation, so overall naming is consistent but not perfectly uniform.

    Tool Count3/5

    With 23 tools, this is on the heavy end for an MCP server and an agent must navigate a large surface. The breadth is defensible for a full LLM evaluation platform covering text, vision, RAG, safety, and PDF benchmarks, but some tools are closely related and could plausibly be consolidated.

    Completeness4/5

    The set covers a full eval lifecycle: case generation, trace ingestion, diverse text/vision/RAG/safety evaluators, report comparison, acceptance policies, and audit packaging. Minor gaps remain—such as no dedicated summarization or code-quality evaluator and no explicit tool for assembling arbitrary eval results into a saved report—but generic G-Eval and custom-rubric tools close most holes.

    Maintenance

    ActivityStale
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      C
      maintenance
      An MCP server that provides AI coding agents with AST-accurate, context-budget-aware codebase querying, safety gates, and team policy integration via structured tools and a local plugin layer.
      121 npm
      4
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      MCP server that enables AI coding agents to communicate, share state, and coordinate work in real time via MCP tools or REST API.
      79 npm
      10
      MIT
    • A
      license
      B
      quality
      D
      maintenance
      MCP server that gives AI coding assistants persistent memory, structural code graph analysis, and safe multi-agent coordination, enabling them to answer architectural questions, track decisions across sessions, and coordinate safely in multi-agent workflows.
      39
      4
      MIT