truth

truth is a deterministic fact-checker for the claims an AI coding agent makes about its own work.

When an agent says "I added the /v1/refund route, set MAX_RETRIES to 5, only touched the parser, and tests pass," truth checks each claim against the real code, the working-tree git diff, recorded command runs, and logs, and answers Supported / Contradicted / Unproven / Refused — every verdict cited.

It is not a chatbot and it does not decide truth with a model. A language model only parses the agent's sentence into a structured claim; a fixed-rule engine decides the verdict from retrieved evidence. The agent cannot talk it into a different answer.

Real commands, real output: the exact verify_turn request an agent sends to truth-mcp and the exact response it gets back. Reproduce it yourself: bash examples/demo.sh.

$ truth verify-turn "I added the /v1/refund endpoint, set MAX_RETRIES to 3, I only changed src/api.rs, and tests pass"

  ✓ Supported     I added the /v1/refund endpoint  (src/api.rs)
  ✗ Contradicted  set MAX_RETRIES to 3             (src/config.rs:1)
  ✗ Contradicted  I only changed src/api.rs        (src/api.rs)
  ! Unproven      tests pass

  1 supported · 2 contradicted · 1 unproven · 0 refused

  ⚠ The agent's message contradicts the evidence above.
  ! You claimed a command succeeded without a recorded run. Run it through
    `truth run -- <cmd>` and re-verify — truth won't confirm an unproven "it passes".

truth won't take "tests pass" on faith. With no recorded run to back it the claim is Unproven — not a shrug, a demand: it blocks the agent's turn (via the Stop hook) until the command is actually run through truth run. Verify a real receipt and it flips to Supported; run it and it fails, and it's Contradicted.

Jump to: Install · Try in 60 seconds · Use from your agent (MCP) · Make it a gate (hooks) · How it works · Benchmarks · Limitations & threat model

Why

Coding agents over-claim. They report success inferred from clean logs, invent terminal output, and confidently describe changes they didn't make. We measured it: across 100 real SWE-agent runs on real GitHub issues, 30% of the attempts that failed the test suite still claimed they fixed the issue ("the method has been successfully added", "the issue has been resolved") — ground truth from the SWE-bench eval, claims judged by an LLM, reproducible.

truth catches the checkable subset of those claims — wrong config values, routes/functions claimed added or removed, files claimed edited, "I only changed X" scope claims, renames, git state ("committed as <sha>", "pushed to origin main"), and "tests pass" — and refuses the genuinely unverifiable rest ("this is cleaner") instead of guessing. A refusal is honest, not a gap: a verifier that bluffs is worse than none.

It also won't accept "tests pass" on faith. A success claim with no recorded run is Unproven — checkable in principle, evidence withheld — and truth demands the proof: it blocks the agent's turn until the command is actually run through truth run. It never runs the command itself (no sandbox, no executing agent code); it makes the agent prove its own claim.

The harder half is the other direction: a verifier that false-accuses gets uninstalled day one. So truth only ever says Contradicted on a structured binary fact — a diff, an AST symbol, an exact value, an exit code — never on a noisy count, and never on a contradiction it parsed out of prose unless that fact backs it. That property is measured, not asserted: a labeled corpus of real agent over-claims holds the false-contradiction rate at 0 as a hard CI gate — and every contradiction truth has ever issued across ~15 of the author's own repos is a regression test. When truth blocks, it means it.

Measured the other way too: replaying 84 real SWE-bench trajectories where the agent claimed success on a task it actually failed, truth caught a file-operation over-claim in 17% of them at zero false accusations — the agent said it created/removed/edited a file its own patch never touched.

Everything runs locally. The store is a single SQLite file in .truth/; raw logs are never persisted (only redacted aggregates); your code never leaves the machine. No LLM, network, or account is required.

Related MCP server: AI Workbench MCP

Install

Claude Code — as a plugin (hooks + MCP tool in one install, all repos):

/plugin marketplace add blasrodri/truth
/plugin install truth@truth

Then install the binary (the plugin will remind you at session start if you skip this):

curl -fsSL https://raw.githubusercontent.com/blasrodri/truth/main/install.sh | sh

The installer downloads the prebuilt tarball and verifies its published SHA-256 before extracting — a corrupted or tampered download fails closed instead of installing. (A trust tool shouldn't ask you to run an unverified binary.)

That's everything. Zero per-repo setup: the hooks fact-check any git repo you work in — the store auto-creates (self-gitignored) on first use, the index auto-refreshes on every check. Opt out of uninitialized repos with truth hook auto off; opt a single repo into project-scoped hooks with truth setup.

Without the plugin — the installer also registers the MCP server when the claude CLI is present; add the hooks once with truth hook install --user (or per repo with truth setup).

Or manually — grab the tarball for your platform from the latest release (macOS arm64/x64, Linux x64/arm64), then:

# verify the download against the .sha256 published next to it, then extract
shasum -a 256 -c truth-*.tar.gz.sha256   # or: sha256sum -c truth-*.tar.gz.sha256
tar -xzf truth-*.tar.gz
install truth-*/truth truth-*/truth-mcp /usr/local/bin/

Or build from source:

cargo build --release --workspace
install target/release/truth target/release/truth-mcp /usr/local/bin/
# binaries: truth (CLI) and truth-mcp (the MCP server)

Why no hosted option? truth is listed on MCP registries (e.g. Glama) for discovery, but it must run on your machine: the verdicts come from your working tree, your git diff, and your recorded test runs — none of which a remote server can see. Local is not a limitation; it's the product (and why your code never leaves the machine).

Try in 60 seconds

Don't take the README's word for it — watch truth catch a lie and reproduce it yourself. The fastest path replays the exact request/response from the GIF above:

bash examples/demo.sh

Or run it by hand against the bundled sample repo (real commands, real verdicts — two true claims and one planted lie):

cp -r examples/sample-repo /tmp/demo && cd /tmp/demo
git init -q . && git add -A && git commit -qm baseline
truth init && truth index .
truth verify-turn "I set the payment retry count to 5, the service runs on \
  port 8080, and I lowered the request timeout to 10 seconds" --repo /tmp/demo

  ✓ Supported     I set the payment retry count to 5  (/tmp/demo/src/routes/checkout.rs:4)
  ✓ Supported     the service runs on port 8080  (/tmp/demo/src/config.toml:3)
  ✗ Contradicted  I lowered the request timeout to 10 seconds  (/tmp/demo/src/routes/checkout.rs:10)

  2 supported · 1 contradicted · 0 refused

The timeout lie is caught against the code; the true claims are Supported with file:line citations. The full step-by-step — scope lies ("I only changed X"), tests pass receipts, and the audit ledger — is in examples/sample-repo/WALKTHROUGH.md.

Audit your own agent history

If you use Claude Code, truth can replay your past sessions and show the over-claims it would have caught — checked against the code as it was at the time (each turn is verified against the commit that was HEAD when the agent wrote it, via a throwaway git worktree, so it never cries wolf about code that moved on):

truth audit-session --last 10            # your 10 most recent sessions
truth audit-session --repo /path/to/repo # just one project

It only audits first-person work reports ("I added X", "tests pass"), not the agent's reasoning or quoted examples — auditing those is a false-positive factory (measured at 81% noise, now filtered out). This is the honest way to see, on your own repos, how often your agent told you something the code contradicts.

Use it from your coding agent (MCP)

truth-mcp is a local Model Context Protocol server (stdio JSON-RPC). It exposes one tool, verify_turn, that an agent calls on its own message before telling you a change is done — so the agent catches and corrects its own lies first.

The repo ships a server.json (MCP registry manifest) and a .mcp.json, so cloning it auto-registers the server in MCP-aware clients (subject to their approval prompt).

Claude Code

truth setup (or install.sh) registers this for you. Manually:

claude mcp add --scope user truth -- truth-mcp
claude mcp list          # verify it connected

Cursor, or any MCP client (generic config)

Add to ~/.cursor/mcp.json (Cursor) or your client's mcpServers block:

{
  "mcpServers": {
    "truth": {
      "type": "stdio",
      "command": "/usr/local/bin/truth-mcp"
    }
  }
}

That's the whole config — no per-repo setup in the MCP file. The agent passes the repo path with each call (the repo argument below), so one registration works across all your projects.

The verify_turn tool

Why claims is the elegant path: truth keeps the hybrid architecture — the LLM parses, the deterministic engine decides. By having the agent (already an LLM, already mid-turn) extract its own claims, truth sidesteps its regex parser entirely for ~tens of tokens, while still never letting a model decide truth. Omissions are caught by the message backstop.

Guaranteeing the agent uses claims. The tool description asks for it, but to make it a habit in your own repos, add a line to your agent's project instructions (e.g. CLAUDE.md):

Before telling me a code change is done, call the `verify_turn` tool: extract
your concrete factual claims (files edited, values set, routes/functions added
or removed, renames, "only changed X", "tests pass") into the `claims` array
and pass the repo path. Run tests through `truth run -- <cmd>` so "tests pass"
is checkable. Fix anything it marks `contradicted` before reporting done.

Or skip the honor system entirely: truth hook install (next section) makes verification run on every turn whether the agent calls the tool or not.

It returns the verdict table as text plus structuredContent (the JSON below), including an index block reporting whether the index is empty or stale — so a "clean" result is never trusted blindly.

One-time per repo: run truth init once to create the .truth/ store. After that, verify_turn auto-refreshes the index on every call — incrementally, skipping unchanged files (~10–50 ms), so code-existence / usage / config claims always reflect the current working tree. You never have to re-run truth index by hand. Claims about the working-tree diff ("I added/removed X") need no index at all. If the index still ends up empty (e.g. nothing indexable), verify_turn says so loudly instead of passing.

Make it a gate the agent can't skip (hooks)

The MCP tool relies on the agent choosing to verify itself. Hooks remove the choice:

truth hook install          # project .claude/settings.json (--user for global)

This registers two Claude Code hooks:

• Stop — when the agent finishes its turn, its final message is fact-checked against the repo, the working-tree diff, and recorded runs. A contradiction blocks the stop and feeds the cited verdict back, so the agent corrects itself before you ever read the claim. An unproven success claim ("tests pass" with no recorded run) also blocks — the agent is sent back to run the command through truth run before it can finish.

• PostToolUse / PostToolUseFailure (Bash) — test/build/lint/fmt commands the agent runs are recorded as command receipts automatically (on success and failure), which is what makes its later "tests pass" checkable — no manual truth run needed. Install just these, without the blocking Stop gate, with truth hook install --receipts (recommended for a global --user install).

Both hooks are fail-open: if truth errors, the session is never wedged. And they're zero-setup: in a git repo that never ran truth init, the hooks auto-create the store at the git root (self-gitignored — your diff stays clean) and start verifying immediately. truth hook auto off restricts them to explicitly initialized repos.

For pull requests, examples/github/pr-factcheck.yml is a drop-in GitHub workflow that fact-checks a PR's description against its actual diff and posts the verdict table as a comment — agent-written PRs get checked before any human reads them.

The lie ledger

Every check is already stored as an audit trail; truth stats reads it back:

$ truth stats --window 7d

  claims checked      142
  supported            96 (68%)
  contradicted          9 (6%)
  refused              37 (26%)
  runs recorded        12 (10 green, 2 failing)

  contradictions by claim type:
    config_value   4
    file_changed   3

truth stats --all aggregates across every repo you've run truth init in (via ~/.truth/registry.json — the stores themselves stay per-repo).

Use it yourself (CLI)

cd your-repo
truth init                      # writes truth.toml + .truth/, runs migrations
truth index .                   # index code/docs/config (re-run after big changes)

truth verify-turn "I added /v1/refund, set MAX_RETRIES to 5, removed /v1/checkout"
truth verify-turn "<agent message>" --repo /path/to/repo --json

truth run -- cargo test         # run + record a receipt; "tests pass" becomes checkable
truth stats                     # the lie ledger

--repo opens that repo's .truth store and diffs that tree. Without it, truth walks up from the current directory to the nearest truth.toml/.truth root (like git does), so every command works from any subdirectory. --json emits stable machine-readable output.

What it can and cannot check

Contradicts only on a structured fact. A Contradicted verdict always rests on a binary fact truth read directly: a file present/absent in the working-tree diff, an AST/index symbol or route, an exact value comparison, or a command exit code. Soft signals — a reference count, a log-occurrence count, a changed-line count — can be wrong (a text scan over-counts comments; a log window samples), so they inform but never accuse: they surface at Inconclusive with their evidence, never as a contradiction. Contradicting a truthful agent on a noisy count is the one failure that gets a verifier uninstalled, so it is structurally impossible, and a labeled corpus gates the false-contradiction rate to 0 in CI (fixtures/precision/, scripts/precision_gate.sh).

Checks (state claims): route added/removed/exists, function/symbol added/removed/exists, config value, named constant ("changed X from 3 to 5" checks the 5), retry count, timeout value, env var present, dependency used, version required, feature-flag enabled — across Rust / TypeScript / Python / Go — against code + git diff + logs, with the diff outranking a possibly-stale index for "I just changed X" claims.

Flags but does not contradict (count-based): "nobody uses X" / "errors are fixed" (log-occurrence counts), "X is unused" (code-reference scan), "updated all 4 call sites" (changed-line count). These surface their evidence at Inconclusive — a flagged suspicion to act on, not a verdict that blocks — because a count can't prove a claim is a lie.

Checks (diff claims — what THIS turn changed): "I edited/created/deleted src/auth.rs" (the diff's file list decides), "I only changed the parser" (catches collateral edits — every changed path must match), "renamed parse_legacy to parse_v2" (old name must be gone AND the new one added). When the tree is clean (work already committed), file-change claims fall back to the HEAD commit: a file that commit touched supports "I created/edited X", and a path that exists nowhere in the repo still contradicts it — so a committed repo isn't a wall of refusals. Locative phrasing ("I added a helper in auth.rs") only asserts the file was touched, not that it was newly created — the change verb is about the helper, not the file.

Checks (command receipts): "tests pass", "go build ./... succeeds", "clippy is clean", "cargo fmt --check passes" — verified against runs recorded by truth run -- cargo test or the Claude Code hook (which now records receipts automatically on both success and failure). Scopes and flags between the command and the result ("cargo test for vllm-parser passes") are handled. Supported only when a matching run exited 0 after your last working-tree edit; a failing receipt contradicts the claim; a green-but-stale receipt proves nothing and is refused.

Checks (git state): "committed as a81b565", "pushed to origin main", "on branch feat/x", "the branch is no longer ahead" — decided from the local object store and remote-tracking refs (no network). A sha that doesn't exist, a branch that doesn't contain it, or an unpushed HEAD contradicts; when there are no remote refs to check a push against, it refuses rather than accuse.

Bilingual extraction: symbol/existence claims are recognized in Spanish too ("checkin.go contiene una función pairAnswersToQuestions"), not just English.

Refuses (by design): action claims with no receipt ("I ran the tests" — record runs and it becomes checkable), judgment claims ("this is cleaner / faster" — no measurable subject), and admissions ("I have not verified X", "I did not run the suite" — an honest disclosure of a gap, nothing to catch). Refused ≠ confirmed. In the JSON output, every claim carries one canonical status — supported / contradicted / partial / refused — and a refused claim adds a refused_reason (not_checkable vs inconclusive) so the "why" is explicit without a second status word. Each report is stamped with the engine_version that produced it.

Configuration

Behavior lives in truth.toml (written by truth init). Tweak it without hand-editing — truth settings validates and preserves the rest of the file, so a user or an agent can change knobs programmatically:

truth settings list                              # every knob, current value, help
truth settings set indexer.extractor mixed       # turn on AST precision (symbols/routes)
truth settings set repo.include src,lib,app      # what to index
truth settings set llm.enabled true              # LLM fallback for phrasings regex can't parse (engine still decides)
truth settings get indexer.extractor --json

The highest-value knob is indexer.extractor: mixed (the default — AST-precise function/struct/route definitions for Rust, TypeScript/JavaScript, Python, and Go, so a symbol named only in a comment isn't mistaken for a real definition; regex fills in everywhere else) · ast · regex (fastest, language-agnostic, noisier). Re-run truth index . after changing it.

How it works

agent message
  → segment into candidate claims (sentences, clauses)
  → claim extraction (deterministic regex first; optional LLM fallback only
     for what regex refuses — never overrides a confident parse, never decides truth)
  → structured claim  (unverifiable → Refused, never guessed)
  → query plan (safe templates only — the LLM never writes LogQL/SQL)
  → evidence: repo index + working-tree git diff (changed lines, file list,
    untracked files) + command receipts (`truth run`) + log queries
  → deterministic verdict engine (fixed rules, source-authority order,
    diff > stale index, receipts must postdate the last edit)
  → cited verdict: Supported / Contradicted / Unproven / Refused
     (Unproven = checkable but the agent withheld the proof, e.g. "tests pass"
      with no recorded run — blocks the turn until the command is actually run)

Every check is stored as an audit trail (the claim, the queries run, the verdict) in SQLite. Log samples are redacted (emails, JWTs, UUIDs, IPs, tokens) before being stored or shown.

Benchmarks

The quality bar is enforced by in-repo gates, not asserted in prose — every number below reproduces with truth eval / scripts/precision_gate.sh:

The number that matters most for a verifier is asymmetric: 0 false contradictions. The precision corpus is built from real agent over-claims (SWE-bench, auto-labeled against each agent's actual patch) plus every prose pattern that has ever false-accused truth on its own repo — and every contradiction truth has ever issued across ~15 real repos of the author's own agent sessions (field_fp_corpus.rs); CI fails the build if any flips to Contradicted.

Recall, measured (the catch side). Replaying 84 real SWE-bench trajectories where the coding agent claimed success on a task it actually failed (benchmarks/swe_overclaim, agent patch applied to the real tree, each concrete claim verified): truth caught a file-operation over-claim in 17% of the failed tasks with 0 false accusations — the agent said it created/removed/edited a file its own patch never touched (e.g. claimed current_config.json was created when the patch made new_config.json). That's a floor, not a ceiling: it counts only file-op lies, the class truth resolves against a hard diff fact; the distilled catches are now a gated regression corpus (recall_corpus.rs) so a "fix" can't silently blind the catcher. Recall on softer claim classes (symbol/relationship) is the open frontier — see docs/BENCHMARKS.md. truth stats --review is the live counterpart: it re-runs past contradictions through the current engine and flags any that no longer fire, self-auditing the lie ledger for false positives an engine fix has retired. Full table (claim types, languages, FP/FN behavior) and reproduction: docs/BENCHMARKS.md.

Limitations & threat model

truth is a tool about trust, so it states its own boundaries plainly. The short version:

• A Contradicted verdict is strong — by construction. It only fires on a structured binary fact (diff presence/absence, AST/index symbol or route, exact value, command exit code); count-based signals can only Inconclusive. And a contradiction truth itself parsed out of an agent's prose (its own segmentation, the imprecise path) is downgraded to Inconclusive unless it rests on such a fact — so a mis-parse can't become a false accusation. The false-contradiction rate is gated to 0 against a labeled corpus of real agent over-claims in CI.

• A Supported verdict is weaker by design — "consistent with the evidence I could read," not "true in every sense." It does not prove the code is correct, only that it matches the claim.

• A Refused verdict is honest, not a gap — for the unverifiable, truth declines to guess. Refused ≠ confirmed.

• The biggest evasion is staying vague. An agent that reports only judgments ("I improved the error handling") is never caught lying because it never said anything falsifiable — though a wall of refusals is itself a signal.

• Everything below the verifier is trusted: the working tree, the git history, the .truth/ store, and the binary itself. A compromised host can defeat any local tool. (This is why install.sh verifies the published SHA-256 before installing.)

The full account — what a verdict can and cannot prove, and a numbered list of how an agent could still evade it — is in docs/THREAT_MODEL.md. For the per-claim-type breakdown of what's checkable, see What it can and cannot check above.

Other commands

truth is built on a general claim/evidence engine; verify-turn is the agent front door. The engine is also usable directly:

check     Check a single natural-language engineering claim
run       Run a command and record a receipt (makes "tests pass" verifiable)
record-run  Record a receipt for a command that already ran (hooks, CI)
stats     The lie ledger: claims checked, contradictions caught, runs recorded
hook      Install agent-harness hooks (verification the agent can't skip)
usage     Observed usage of a route/event/pattern (deterministic)
errors    Error occurrences (deterministic)
config    Search indexed config/code definitions
owners    Who has worked on the code behind a subject
uses      Find code references to a symbol/route/dependency
docs      Is a subject documented, and consistent with code?
inspect   Show exactly what was indexed (trust the evidence)
doctor    Validate local setup and explain readiness
claims/report/ci/eval/diff   claim files, reports, CI gates, regression diffs

Run truth <command> --help for details. truth serve (Slack/HTTP) is an informational placeholder and intentionally not built — the local verifier is the product.

Tests

cargo test --workspace

Covers extractors (Rust/TS/Python/Go routes, constants, env vars, deps, file/ scope/rename/count/command claims), the git-diff adapter (changed lines, file statuses, untracked files), the verdict rules and golden fixtures (including receipt freshness: green-but-stale runs never pass), claim segmentation, hook settings merging, index-freshness warnings, JSON output, and end-to-end checks over the sample repo. truth eval fixtures/eval/agent_claims.yaml is the agent-fact-checking quality harness.

Measuring extraction quality

fixtures/eval/extractor_corpus.yaml is a diagnostic corpus, not a gate: the same ground-truth facts phrased many ways, including hard H* edge cases the regex extractor is expected to miss. Run it to measure where claim extraction stands and what a better extractor (agent-supplied claims, a local LLM, or AST) would improve:

truth eval fixtures/eval/extractor_corpus.yaml

A T*/H* case that returns inconclusive is a recall gap (extractor too weak); an F* case that returns supported is a dangerous false pass; an R* case that returns a verdict is a hallucination. The bands make all three visible, so changes can be measured instead of guessed.