Hallucination Firewall

An MCP server that checks an AI agent's final answer against the tool outputs it actually received, and blocks or flags anything unsupported.

Agents call tools, get correct data back, and still misstate it in the answer: a wrong figure, an invented detail, a claim no source supports. The Hallucination Firewall records everything the tools returned, splits the draft answer into single-fact claims, checks each one against that evidence with a tiered verifier, and returns a verdict per claim with the tool output that proves or disproves it.

https://github.com/Parvathy-Raj/hallucination-firewall/raw/main/docs/demo.mp4

Demo: Claude Code answers a UAE VAT question using a document-search tool. The draft gets the registration threshold wrong; the firewall flags it ❌, cites the FTA guide passage with the correct figure, and the corrected answer goes out with a citation.

┌──────────┐     ┌────────────────────────────┐     ┌──────────────────┐
│  Agent   │────▶│   Hallucination Firewall   │────▶│ Downstream MCP   │
│ (Claude) │◀────│   (MCP server + client)    │◀────│ servers (DB, RAG,│
└──────────┘     │                            │     │ web, files...)   │
                 │  1. Evidence ledger        │     └──────────────────┘
                 │  2. Claim extractor        │
                 │  3. Evidence retriever     │
                 │  4. Verifier (tiered)      │
                 │  5. Verdict + rewrite      │
                 └────────────────────────────┘

How it works

1. Evidence ledger. Every tool output is recorded, chunked and indexed with hybrid retrieval (BM25 + bge-small embeddings, merged with reciprocal rank fusion). Outputs get there three ways:

   • Proxy mode: the firewall wraps your other MCP servers and re-exposes their tools (uae_docs__search_guides), recording every result.

   • Claude Code hook: a PostToolUse hook records the output of any tool (Read, WebFetch, Bash, other MCP servers).

   • By hand / SDK: record_evidence(source, content), or fw.record() / @fw.watch in Python.

2. Claim extraction. An LLM splits the draft into atomic claims and labels each one factual, opinion or hedge. Only factual claims are checked. (A rule-based extractor is used when no LLM is configured.)

3. Tiered verification, cheapest first. Each tier either decides or passes the claim on:

   Tier	Method	Cost	Catches
   0	Exact checks: numbers, currency, percentages, dates, names, with rounding-aware matching	~0	$4.2M vs $4.7M, AED 500,000 vs AED 375,000
   1	NLI fact-checking model on CPU (DeBERTa-v3-base-mnli-fever-anli)	~0.15 s per evidence passage	Most clearly supported or contradicted claims
   2	LLM judge, only when tier 1 is below the confidence threshold	$	Paraphrases, multi-hop, "the evidence doesn't say that"

4. Verdict + policy. Each claim is ✅ supported, ❌ contradicted, ⚠️ unsupported or ➖ not_checkable. The policy decides whether to block, annotate or rewrite. The rewrite is deterministic on purpose (an LLM rewriter could add hallucinations of its own): wrong figures are replaced with the evidence value, unsupported claims are hedged or removed, and citations to the tool outputs are added.

Related MCP server: footnote

The trust model (read this first)

An MCP server cannot force an agent to call verify_answer. This project closes that gap in three layers, from weakest to strongest:

What it does not do: check claims against world knowledge. A claim that no tool output supports is unsupported, even if it happens to be true. That is the point: in an agent, an answer should be traceable to what the tools returned.

MCP interface

Tools

Resources: ledger://session/current (everything the tools returned) and report://session/current (hallucination report for the session).

Quickstart

git clone https://github.com/Parvathy-Raj/hallucination-firewall
cd hallucination-firewall
uv sync --extra ml          # tiers 1 and dense retrieval (CPU PyTorch); plain `uv sync` = tier 0 + BM25 only
uv run pytest -q

Check a draft from the command line:

uv run hfw verify --evidence report.txt --draft draft.txt --rewrite

Use it in Claude Code

1. Copy examples/claude_code/.mcp.json into your project and fix the paths. It runs the firewall in proxy mode in front of the example UAE tax-docs server (downstream.json); put your own MCP servers there.

2. Copy the hooks from examples/claude_code/settings.json into .claude/settings.json. capture records every tool output; stop makes Claude verify before it answers.

3. Set an LLM for claim extraction and the tier-2 judge (below). Without one, tiers 0 and 1 still run.

Claude Desktop: see examples/claude_desktop/claude_desktop_config.json.

Use it from Python (enforced in code)

from hfw import Firewall, HallucinationBlocked

fw = Firewall()

@fw.watch                      # every return value becomes evidence
def get_invoice(invoice_id: str) -> dict: ...

get_invoice(invoice_id="INV-1042")
try:
    answer = fw.guard(draft)   # raises if the policy blocks
except HallucinationBlocked as e:
    answer, _ = fw.rewrite_grounded(draft)

Full example: examples/sdk_example.py.

Configuration

Evaluation

Three benchmarks, all scored at the claim level (one claim per answer sentence, so human span labels map onto claims directly and every configuration sees identical claims):

• RAGTruth (main benchmark): RAG answers (QA, summarization, data-to-text) with human-labelled hallucinated spans. Test split, sampled.

• HaluEval QA: each question once with the right answer and once with a hallucinated one.

• UAE agent traces (custom): 20 simulated agent turns built from the FTA guide chunks of my UAE regulatory RAG. The tool returns the gold passage plus two distractors; the draft has one correct sentence and three planted errors (changed number, negated obligation, claim from a guide that was never retrieved). Rebuild with make traces UAE_CHUNKS=...; the corpus itself is not redistributed.

Metrics: precision / recall / F1 for catching hallucinated claims; false-flag rate (correct claims wrongly flagged); false-block rate (fully correct answers that the balanced policy would block, which is what users notice); latency p50/p95 per answer; LLM cost per answer.

make eval regenerates everything below. Every NLI score and LLM call is cached, so the configurations share identical judge calls and re-runs are free; cost and latency still charge each configuration for every call it makes.

Run make eval to fill this in.

Reading the results

The numbers here come from a small sample (rate limits on the free Groq tier I used for the judge), so treat differences of a few points as noise. The judge in these runs is gpt-oss-20b via Groq, not Claude; set HFW_EVAL_MODEL / HFW_EVAL_BASE_URL to compare others.

Repo layout

src/hfw/
├── server.py        MCP server + tools + resources
├── proxy.py         downstream MCP clients, ledger capture
├── ledger.py        evidence store, chunking, hybrid index
├── claims.py        claim extraction (LLM or rules)
├── verify/
│   ├── exact.py     tier 0
│   ├── nli.py       tier 1
│   ├── judge.py     tier 2
│   └── pipeline.py  confidence routing
├── policy.py        strictness presets, block / annotate / rewrite
├── rewrite.py       deterministic grounded rewrite with citations
├── firewall.py      facade used by the server, CLI and SDK
├── hooks.py         Claude Code PostToolUse + Stop hooks
└── cli.py           `hfw serve`, `hfw verify`
evals/               datasets, runner, custom traces, results/
examples/            Claude Code / Desktop configs, demo server, SDK example
tests/               pytest (offline, no models needed)

Likely questions

Roadmap

• MiniCheck / Vectara HHEM as alternative tier-1 models, and a FEVER run

• Claim-to-span alignment so rewrites can edit inside long sentences

• OpenTelemetry spans per tier

• PyPI release

License

MIT