Skip to main content
Glama

Groundcheck, verification with no model in the loop

ci python license DOI

I built this MCP connector to check whether the grounding behind a claim is real. Is the quote actually on the page? Does the arXiv id resolve? Does the code print what someone says it prints, and is the number right? There's no language model anywhere in the verification path. It works in any MCP host, so Claude, Gemini or something else. The checkers under verify/ re-derive every verdict it returns. They run against the same fixtures and don't share the server's code. If a re-derivation drifts, the build stops.


Abstract

When an assistant "fact-checks" something, it almost always asks a second model whether the first one was right. That just moves the error somewhere else, since the checker hallucinates too. I wanted to check grounding against reality instead. Groundcheck has five tools. They fetch a page, resolve an identifier, run a snippet, grep a codebase or evaluate an expression. Each one returns one of three verdicts with the evidence attached.

I kept the scope narrow on purpose. Groundcheck confirms that the evidence a claim rests on is real and says what it's quoted to say. It doesn't judge whether a claim is semantically true. "This quote is on the cited page" can be checked. "The page's argument is correct" can't. When it can't tell, it returns unverifiable.

No language model is involved in any verdict.

Contributions. (i) Verification grounded in sources. (ii) A three-verdict contract with an explicit unverifiable, so refusing to answer is a proper outcome. (iii) Results you can audit, since every verdict carries the quote, stdout, matching line or computed value it was based on.


Related MCP server: math-logic-mcp

1. Why this, and why it's hard

Almost every "fact-check" built into an assistant today ends up asking a second model whether the first one was right. That doesn't really verify anything. The checker hallucinates too, so the error just ends up in a different place. Checking against reality instead of another model's opinion is the hard part, and few tools attempt it. That's the only thing this one does.

I want to be careful about scope, since over-claiming would undercut the whole idea. Groundcheck confirms that the evidence a claim rests on is real and says what it's quoted to say. It doesn't judge whether a claim is semantically true. "This quote is on the cited page" is checkable. "The page's argument is correct" isn't, and pretending won't change that. Every tool returns one of three verdicts. When it can't tell, it says unverifiable.

verdict

meaning

checked

the grounding was confirmed against a real source

refuted

the source exists and contradicts the claim (wrong number, missing quote, dead id, failing code)

unverifiable

no source, or it needs judgement this tool refuses to fake

flowchart LR
    A["Assistant makes<br/>a claim"] --> M{"Groundcheck<br/>MCP server"}

    M --> Q["check_quote<br/><i>fetch the page</i>"]
    M --> C["check_citation<br/><i>query arXiv / Crossref</i>"]
    M --> X["check_code<br/><i>run in a subprocess</i>"]
    M --> R["check_repo<br/><i>grep the files</i>"]
    M --> T["check_math<br/><i>evaluate an AST</i>"]

    Q & C --> NET[("the live web")]
    X & R --> LOCAL[("local filesystem<br/>and interpreter")]
    T --> PURE[("arithmetic")]

    NET & LOCAL & PURE --> V{"verdict"}
    V --> OK["checked"]
    V --> NO["refuted"]
    V --> UNK["unverifiable"]

    classDef src fill:#4d4d4d,stroke:#2b2b2b,color:#fff
    classDef good fill:#1a9850,stroke:#0f6b33,color:#fff
    classDef bad fill:#b2182b,stroke:#7f0f20,color:#fff
    classDef warn fill:#f4a582,stroke:#c06a4f,color:#000
    class NET,LOCAL,PURE src
    class OK good
    class NO bad
    class UNK warn

None of the boxes in that diagram is a language model. Every verdict ends at something you can look up, run or compute.

2. The tools

tool

verifies

how (no LLM)

check_quote(quote, url)

an exact quote is on a page

fetch the page, match the text

check_citation(identifier)

an arXiv id or DOI resolves

query arXiv / Crossref, return the real title

check_code(snippet, expected_output)

code prints what's claimed

run it in a subprocess, compare stdout

check_repo(pattern, path)

a string/regex is in a codebase

grep the files, return real matching lines

check_math(expression, claimed_result)

arithmetic is correct

evaluate an AST (noeval), compare

Every result has the shape {status, method, evidence, detail}. The evidence field holds whatever was actually found, like the quote, the stdout, the matching line or the computed value. That's what makes a verdict auditable.

2.1 A live run

real verdicts from a live run

Each row above is a real call to the same function the server exposes. That includes the arXiv lookups that need the network. The refutations are real too. The first row is the 3.7 x 1400 error from section 3, reproduced.

2.2 The case corpus

verify/export_cases.py writes these tables. I keep them tracked in git so CI can corrupt one and make sure the harness notices.

table

cases

checked

refuted

unverifiable

cases_math.tsv

27

17

3

7

cases_repo.tsv

10

6

2

2

cases_text.tsv

11

6

4

1

verdict split of the case tables

The same three tables as bars, counted from the TSV files. Every table has cases for all three verdicts.

how verify/verify.sh recomputes the verdicts

verify/verify.sh recomputes these verdicts in SQL, C, Go, R, Rust and Node, without sharing the server's code. The build stops if any of them disagrees.

3. It caught a mistake in its own author's work

I wrote check_citation because made-up arXiv ids that looked plausible kept slipping into research write-ups. They look right and resolve to nothing. check_math exists because 3.7 × 1400 was written as 8880 in a hardware deck. It's actually 5180. check_repo grew out of a claim-checker for my profile README that compares every quoted number with the repo it came from. So each tool started as a mistake that really happened.

One thing surprised me. While testing, I assumed arXiv 2606.01992 was fabricated and expected refuted. The tool returned checked. It was right and I was wrong, because it's a real June-2026 paper. The verifier caught my own bad assumption, and that's exactly why I wanted verification grounded in a source.

4. Use it

pip install -e .          # or: pip install -r requirements.txt
python -m pytest tests/   # 23 tests, no network needed (mocked transport)

For Claude or Claude Code, add this to your MCP config.

{
  "mcpServers": {
    "groundcheck": { "command": "python", "args": ["-m", "src.groundcheck.server"] }
  }
}

For Gemini CLI or any other MCP host, it's the same stdio server. Point your host's MCP config at python -m src.groundcheck.server. MCP is why one connector can serve both.

5. Security

check_code runs the code you give it in a subprocess. It uses list-form subprocess, so there's no shell and nothing to inject, and it kills the process on timeout. It isn't sandboxed from the network or the filesystem, though. Only pass code you'd run yourself. The other four tools are read-only (HTTP GET, file read, arithmetic).

6. Limitations

This checks grounding. It doesn't check truth, and that's by design (see Scope above).

Quote matching is exact, apart from normalising whitespace. A paraphrase that means the same thing returns refuted. Deciding that two sentences "mean the same" needs a judge, and this tool refuses to be one. Match the literal text.

check_quote reads the served HTML, so it won't find a quote on a JS-rendered page that client-side JavaScript injects. It fails safe with refuted and never gives a false checked.

Citations only go through arXiv and Crossref. Other registries aren't wired up yet.

7. Licence

MIT, see LICENSE.

Available Tools

5 tools
check_citationA

Verify that an arXiv id or DOI actually resolves, and return its real title.

Pass an arXiv id (e.g. '2412.18004' or 'arXiv:2412.18004') or a DOI. A 'checked' verdict includes the genuine title, so a plausible-looking but fabricated identifier is caught as 'refuted'.

Inputs: identifier (required).

check_codeA

Run a self-contained Python snippet and confirm its stdout contains expected_output.

WARNING: this executes the code. Only pass code you would run yourself; it is not sandboxed from the network or filesystem. Returns refuted if the snippet errors or prints something other than claimed.

Inputs: snippet (required), expected_output (required).

check_mathA

Confirm an arithmetic expression equals a claimed result.

Evaluates expression (numeric literals and + - * / // % ** only) and compares to claimed_result. Catches arithmetic a model asserted without computing.

Inputs: expression (required), claimed_result (required).

check_quoteA

Verify that an exact quote appears on a web page.

Fetches url and confirms quote is present in its visible text. Use before attributing a statement to a source. Returns refuted if the page loads but the quote is absent -- i.e. the attribution is fabricated.

Inputs: url (required), quote (required).

check_repoA

Confirm that a string (or regex) actually occurs in files under a local path.

Use to verify a claim about what a codebase or document contains -- 'the README reports 0.9086', 'the function is defined in src/' -- rather than trusting the assertion. Returns the real matching lines.

Inputs: path (required), pattern (required), regex.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.2/5.0

    Scored across 5 tools

    Disambiguation5/5

    Each tool verifies a distinct artifact type: web-page quote, arXiv/DOI identifier, Python snippet output, local repo content, and arithmetic. The input types and returned verdicts differ clearly, so an agent can pick the right tool without overlap.

    Naming Consistency5/5

    All five tools follow an identical check_<noun> pattern (check_quote, check_citation, check_code, check_repo, check_math). This is a fully predictable, uniform convention.

    Tool Count5/5

    Five tools is well-scoped for a claim-verification server, with each tool earning its place by covering a different source of unverified assertions. No redundancy or filler.

    Completeness4/5

    The surface covers the main verification targets (quotes, citations, code, repo contents, math) and returns actionable verdicts. Minor gaps exist, such as no standalone URL-liveness check or batch/multi-source verification, but core workflows are covered.

    Maintenance

    ActivityNo data
    ResponsivenessOutdated data

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      B
      maintenance
      MCP server for verifying AI agent claims vs reality — single-transcript inline grounding-check that flags when an agent's response states facts not in the input context, when its code silently swallows exceptions and substitutes mock data, or when its multi-turn transcript contains contradictions or unverified completion claims. Sub-second, local, free, no API calls.
      4
      24 PyPI
      1
      MIT
    • A
      license
      Not graded
      quality
      D
      maintenance
      An MCP server that verifies whether a claim is actually supported by the source text at a given citation — independent of what the calling LLM asserts.
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Provides an MCP server for storing and querying knowledge as verifiable claims, enforcing evidence-backed assertions with exact quotes and refusing paraphrases or unsupported relations.
      Apache 2.0