Skip to main content
Glama

TruthGate (codesphere-u/truthgate)

CI Python 3.11+ License: MIT MCP Compatible

Deterministic Agent Hallucination Auditor, Citation-Span Entailment Verifier & Workspace Symbol Grounding Engine (with MCP Server)


Why TruthGate?

Even when AI agents use RAG (cortex-mcp, auralis-local) or codebase search tools, they frequently produce subtle, high-severity hallucinations:

  1. Phantom Citations & Unsupported Claims (TG001, TG002): The agent appends [doc_99] to a claim or attributes a fabricated statement to [sec_policy].

  2. Negation & Antonym Polarity Flip (TG003): The source says "TLS 1.0 is not permitted in production", and the agent summarizes "TLS 1.0 is permitted in production [sec_policy]".

  3. Numeric, Percentage & Currency Drift (TG004): The tool output reports latency = 42.8 ms or UGX 1,250,000, and the agent writes 4.28 ms or UGX 125,000.

  4. Hallucinated Code Symbols, File Paths & Line Ranges (TG006–TG008): Coding agents reference [src/auth/jwt.py#L120-L150] when jwt.py only has 24 lines, or call verify_token_v2() when the function does not exist in the workspace AST.

  5. Expensive LLM-as-a-Judge: Calling a second frontier LLM to audit every response adds 3 seconds of latency and doubles API spend. TruthGate provides a sub-millisecond (0.12 ms P50), zero-dependency deterministic verification gate that catches citation mismatches, polarity inversions, numeric drift, and phantom workspace references—and compiles a surgical self-correction prompt when a check fails.


Related MCP server: flameox

Architecture

flowchart LR
    Resp["Agent Markdown Response"] --> Decomp["ClaimDecomposer\n(Atomic Claims, Citations,\nNumbers, Negations, CodeRefs)"]
    Decomp --> Entail["CitationEntailmentVerifier\n(Rules TG001–TG005)"]
    Decomp --> Ground["WorkspaceSymbolGrounder\n(Rules TG006–TG008)"]
    Docs[("SourceDocument Corpus")] --> Entail
    AST[("Workspace Files & AST")] --> Ground
    Entail --> Gate{"Verification\nPassed?"}
    Ground --> Gate
    Gate -- "True" --> Pass["Deliver Grounded Response"]
    Gate -- "False" --> Repair["SurgicalRepairCompiler\n(Self-Correction Prompt)"]

Deterministic Verification Rules (TG001–TG008)

Rule Code

Name

Category

Verdict Status

Severity

Deterministic Check

TG001

Phantom Source Citation

Entailment

PHANTOM_CITATION

CRITICAL

Claim cites a [source_id] that does not exist in the provided context corpus.

TG002

Low Lexical/Entity Support

Entailment

UNVERIFIED

HIGH

IDF-weighted stem and technical-entity overlap between claim and cited source falls below threshold.

TG003

Negation Polarity Inversion

Entailment

CONTRADICTED

CRITICAL

High entity overlap with a source sentence, but opposite negation or antonym polarity (enabled/disabled, permitted/prohibited, encrypted/unencrypted, immutable/mutable, synchronous/asynchronous).

TG004

Numeric & Magnitude Drift

Entailment

NUMERIC_DRIFT

CRITICAL

Claim introduces a number, percentage, or currency (UGX, USD, EUR, ms, %) that does not match any value in the cited source.

TG005

Uncited Factual Assertion

Entailment

UNVERIFIED

MEDIUM

Factual or numeric proposition in strict-grounding mode lacks any supporting [source_id] citation or workspace evidence.

TG006

Non-Existent File Path

Workspace Grounding

BROKEN_CODE_REF

CRITICAL

Referenced file path (src/missing.py) does not exist in workspace_root (or attempts path traversal).

TG007

Out-of-Bounds Line Range

Workspace Grounding

BROKEN_CODE_REF

HIGH

Referenced line range (file.py#L120-L150) exceeds the actual line count of file.py.

TG008

Phantom Code Symbol

Workspace Grounding

BROKEN_CODE_REF

CRITICAL

Referenced function, class, or method (verify_token_v2()) does not exist in the target file's Python AST or TS/JS/Go/Rust symbol table.


Verified Benchmark Results (40-Scenario Suite)

Reproduce locally via truthgate benchmark:

Metric

Verified Value

Notes

Total Scenarios

40

10 Grounded Truthful + 30 Adversarial Hallucinations (TG001–TG008)

Precision

100.0%

30 / 30 flagged responses are true hallucinations

Recall

100.0%

30 / 30 adversarial hallucinations caught across all 8 rules

False Positive Rate (FPR)

0.0%

0 / 10 truthful RAG & coding responses falsely flagged

Exact Rule Code Accuracy

100.0%

40 / 40 scenarios match exact expected TG001–TG008 rule code

P50 Verification Latency

0.12 ms

Sub-millisecond deterministic execution

P95 Verification Latency

0.37 ms

< 0.5 ms tail latency (10,000x faster than LLM-as-a-Judge)


Quickstart

Installation

git clone https://github.com/codesphere-u/truthgate.git
cd truthgate
python3 -m pip install -e ".[dev]"

Run the End-to-End Audit & Self-Correction Demo

truthgate demo

Output excerpt:

[Audit Summary] Passed: False | Score: 0.25 | Claims: 1/4 supported | Latency: 1.207 ms

[Claim-by-Claim Verdicts]
  - [FAIL:TG003] (CONTRADICTED) TLS 1.0 is permitted in production environments [sec_policy].
  - [FAIL:TG004] (NUMERIC_DRIFT) The P95 gateway routing latency is 4.28 ms across East Africa edge nodes [sla_metrics].
  - [PASS] (SUPPORTED) Daily instant settlement volume is capped at UGX 1,250,000 for Tier-1 merchant accounts [fin_ledger].
  - [FAIL:TG008] (BROKEN_CODE_REF) In `src/auth/jwt.py`, call `verify_token_v2()` to validate bearer tokens.

[Compiled Surgical Self-Correction Prompt]
------------------------------------------------------------------------------------
[TRUTHGATE VERIFICATION FAILED — 3 violation(s) detected: TG003, TG004, TG008]
Rewrite your response to fix ONLY the following deterministic verification failures:
1. [TG003 (Negation Polarity Inversion)] Offending statement: "TLS 1.0 is permitted in production environments [sec_policy]."
   - Ground-Truth Evidence: "TLS 1.0 is not permitted in production environments."
...

Python SDK Usage

from truthgate import SourceDocument, TruthGateVerifier

sources = [
    SourceDocument(
        source_id="sec_policy",
        title="Production TLS Policy",
        content="TLS 1.0 is not permitted in production environments.",
    ),
    SourceDocument(
        source_id="sla_metrics",
        title="Gateway Telemetry",
        content="The P95 gateway routing latency is 42.8 ms across East Africa edge nodes.",
    ),
]

verifier = TruthGateVerifier()
result = verifier.verify(
    response_text=(
        "TLS 1.0 is permitted in production environments [sec_policy]. "
        "The P95 gateway routing latency is 4.28 ms across East Africa edge nodes [sla_metrics]."
    ),
    sources=sources,
)

print("Passed:", result.passed)
print("Violations:", result.violation_counts)  # {'TG003': 1, 'TG004': 1}
print(result.repair_prompt)

CLI Reference

# 1. Run end-to-end hallucination audit & repair prompt demo
truthgate demo

# 2. Run the 40-scenario benchmark suite (human-readable or JSON)
truthgate benchmark
truthgate benchmark --json

# 3. List all 8 deterministic verification rules (TG001-TG008)
truthgate rules

# 4. Verify an agent response against a JSON sources file and workspace directory
truthgate verify \
  --response "TLS 1.0 is not permitted in production environments [sec_policy]." \
  --sources-json sources.json \
  --workspace . \
  --fail-on-violation

# 5. Ground workspace file paths, #Lstart-Lend line ranges, and AST symbols
truthgate ground \
  --response "See [models.py](truthgate/models.py#L1-L40) and \`TruthGateVerifier\`." \
  --workspace .

# 6. Start the JSON-RPC 2.0 Stdio MCP server
truthgate mcp

MCP Server Integration

Add truthgate to your Claude Desktop, Cursor, or Antigravity mcp_config.json:

{
  "mcpServers": {
    "truthgate": {
      "command": "truthgate",
      "args": ["mcp"]
    }
  }
}

Exposed MCP Tools

  1. truthgate_verify_response: Verifies an agent response against source documents and an optional workspace directory, returning claim-by-claim verdicts and a surgical repair prompt.

  2. truthgate_ground_workspace_refs: Verifies all file paths, #Lstart-Lend line ranges, and AST symbols referenced in markdown text against a workspace directory.

  3. truthgate_list_rules: Lists all 8 deterministic verification rules (TG001–TG008).

  4. truthgate_run_benchmark: Runs the reproducible 40-scenario benchmark suite.


Part of the Codesphere Agentic Infrastructure Stack

Built by Akandwanaho Ivan Alvin (@codesphere-u), Founder & MD of Codesphere Innovations (U) Limited in Kampala, Uganda:

License

MIT License — see LICENSE.

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    C
    maintenance
    Provides deterministic AST graph queries and rule validation for AI coding agents via MCP tools, enabling structural intelligence without LLM calls.
    3 npm
    MIT
  • A
    license
    C
    quality
    B
    maintenance
    Enables coding agents to query, compare, and audit local profiler traces, benchmarks, memory captures, and execution evidence without uploading code or data, using CLI and MCP interfaces.
    111
    88 PyPI
    232
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables AI coding agents and hosts to enforce deterministic repository boundaries via MCP, providing structured reads, supervised edits, snapshots, audits, and recovery with machine-readable evidence.
    MIT