groundcheck

Playwright verifies your UI. Sentry verifies your runtime. groundcheck verifies your AI's answers.

An MCP server that lets any AI agent evaluate RAG outputs -- faithfulness scoring, hallucination detection, and retrieval quality metrics -- with zero API keys, using MCP sampling so the judge model is whatever the connected client is already running.

The problem

You've vibe-coded a RAG app. It answers, but sometimes it makes stuff up, and you can't tell when. There's no test for "did the model lie about what the documents say." groundcheck is that test.

Related MCP server: multivon-mcp

Demo

Eval dashboard

groundcheck-dashboard --out docs/dashboard.html generates a single self-contained HTML file from your report store -- summary cards, a faithfulness-by-report chart, a sortable per-case table, and worst-5-cases, all with Chart.js loaded from a CDN and the data embedded inline. Opens straight from disk, no server required.

docs/dashboard.html in this repo is generated from a real report: knowledge-assistant's 18-query eval (see its Evaluation section, cross-linked above) -- not synthetic smoke-test numbers. Enable GitHub Pages on this repo to get a live, clickable link instead of a clone-and-open file (see Roadmap).

Quickstart

Claude Code:

claude mcp add groundcheck -- uvx groundcheck

Claude Desktop -- add to claude_desktop_config.json (full example):

{ "mcpServers": { "groundcheck": { "command": "uvx", "args": ["groundcheck"] } } }

Cursor -- same shape, in .cursor/mcp.json (see examples/claude_code.md for the full config).

No API key needed if your client supports MCP sampling. Try it: ask your agent to run groundcheck_detect_hallucinations on an answer and its sources.

Tools

How it works

flowchart LR
    subgraph Client["Your MCP client (Claude Desktop / Code / Cursor)"]
        Model[("Your LLM")]
    end

    Client -- "tool call" --> Server["groundcheck MCP server"]

    subgraph Server["groundcheck"]
        Det["Deterministic tools\nmetrics.py -- pure Python\nprecision@k, recall@k, MRR, NDCG"]
        Judged["LLM-judged tools\nclaim decomposition + verification"]
    end

    Judged -- "sampling/createMessage" --> Model
    Model -- "judged verdicts" --> Judged

    Server -- "result" --> Client

The split matters: retrieval metrics with gold labels are pure math and run instantly with zero model calls. Faithfulness, hallucination detection, and compare need semantic judgment, so they call back into your own connected model via MCP sampling -- no separate API key, no separate bill. If your client doesn't support sampling yet, set ANTHROPIC_API_KEY as a fallback; if neither is available, you get a clear error naming both options.

Cost: deterministic metrics are free (no model calls). LLM-judged tools cost 1-2 model calls via your client's existing inference -- no API key required on top of what you're already paying your client for.

What groundcheck is NOT

• Not an observability platform. It doesn't collect traces, dashboards, or alerts over time -- it scores the RAG output you hand it, once, when you ask. For production observability, look at LangSmith or Arize Phoenix.

• Not for agent-trajectory evals. It judges answers against sources, not whether an agent took the right sequence of actions.

• Not enterprise-scale. Reports are local JSON files. If you need multi-tenant dashboards, RBAC, or dataset versioning at scale, LangSmith or Phoenix are the right tool.

Case studies

• evals/RESULTS.md tracks tool-selection and hallucination-detection accuracy before and after tuning tool docstrings -- the actual before/after numbers, not just the final descriptions.

• Real RAG app: knowledge-assistant (a multi-tenant document Q&A app) uses groundcheck to score its own real pipeline output -- 18 real queries against a real document, 85% mean faithfulness, 94% mean NDCG@5. The interesting finding wasn't a perfect score: the app never hallucinated a fact, but groundcheck's strict literal-support judge flagged several accurate paraphrases as "unsupported" -- a useful data point about judge strictness, not just app quality. Full breakdown in its README Evaluation section.

Security

Read-only, compute-only server: no shell access, no network egress except an opt-in ANTHROPIC_API_KEY fallback, no filesystem writes outside the local report store. dataset_path is validated against an allowlisted directory to prevent path traversal. Full threat model in SECURITY.md.

Roadmap

• MCP Tasks primitive for async run_suite on large datasets.

• MCP Apps report UI (render a report inline instead of raw JSON).

• Publish to PyPI and the MCP Registry (packaging is ready; not yet published).

• Live FastAPI dashboard (v2) -- groundcheck-dashboard's static HTML file is the v1 by design (zero deployment burden); a running app with live updates is a reasonable next step once there's a reason to keep one up.

• Enable GitHub Pages for docs/dashboard.html so there's a live, clickable "see eval results" link with no clone required.

Development

uv sync --all-extras
uv run pytest
uv run ruff check .

MIT licensed. See LICENSE.