agenttruth

CI check that fails when CLAUDE.md / AGENTS.md / README stop matching the code they describe.

Your agents read your instruction files on every run. If those files lie, every run is wrong — and nothing tells you, because docs don't fail builds.

npx agenttruth

agenttruth · 2 docs · 468 files indexed
8 drift findings in 1 file
README.md
    878  missing-file   references "policy.json" — no such file in repo
    904  missing-file   references "agent-browser.json" — no such file in repo

Exit code 1 — your CI fails. That's the point.

Why

The 2026 agent-tooling wave is very good at feeding agents context and very bad at checking that context is true. graphify, codegraph, archify, CLAUDE.md generators — dozens of projects building knowledge graphs and instruction files. Zero of them verify the output still matches the code. So an agent reads a stale CLAUDE.md and confidently does the wrong thing.

agenttruth is the missing half: the CI gate that fails on drift.

Related MCP server: tamperlens-mcp

Install

npx agenttruth              # scan the current repo
npm i -D agenttruth         # or add to devDependencies

Requires Node 18+. Zero runtime dependencies.

Checks

numeric-claim — a count in your docs that the code contradicts.

README: "covered by a 299-check behavior harness" → repo has 266 assertions.

Counters are matched by kind, so a test count is never compared to an agent count. Only typo-sized differences are flagged (a stale count from an older commit), never unrelated numbers. Helper declarations (function check()) are excluded.

missing-file — docs name a file the repo doesn't ship.

CONTRIBUTING: "see e2e_mcp.rs for the integration harness" → file does not exist.

It stays quiet on the things that aren't drift: badge URLs, CLAUDE.md files you tell people not to use, files a CLI command expects the user to supply (--params @input.json), submodule paths, and anything inside a fenced code block.

broken-link — a relative markdown link that resolves to nothing.

README: "see the Authentication docs" → file does not exist.

Also catches directory links with no index and anchors with no matching heading. Absolute http(s) links are skipped on purpose — verifying them would make CI depend on the network.

missing-command — docs tell you to run a build tool the repo doesn't have.

README: "make test" → no Makefile, makefile, GNUmakefile, Justfile, or justfile.

Only counts commands in shell-fenced blocks, $-prefixed lines, or indented snippets. Prose like "installation via npm is deprecated" or "a fix that doesn't make sense" is never flagged.

Auto-fix

npx agenttruth --fix

Rewrites stale counts to the measured value. Only unambiguous ones — if two files claim to hold "the agent catalog" and disagree, it refuses rather than guesses, because a wrong number in a README is worse than a stale one.

Severity

npx agenttruth --fail-on error    # block the build on errors only
npx agenttruth --fail-on warning  # errors and warnings

Inline disables

Every finding can be waved through in place, with a reason. All invisible in rendered markdown.

<!-- agenttruth-disable-next-line reason: Makefile is vendored at build time -->
```bash
make test

<!-- agenttruth-disable-line -->       this line only
<!-- agenttruth-disable-file -->        the whole file

disable-next-line covers an entire fenced block when the next line opens one.

GitHub code scanning

- run: npx agenttruth --sarif agenttruth.sarif --fail-on none
- uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: agenttruth.sarif

Findings then annotate the exact file and line on the pull request, next to the diff that caused them — rather than scrolling past in a CI log.

Speed

Counters are measured once per repo, not once per document. Before memoisation a 5,873-file repo with 42 markdown files took 4m34s — nobody waits that long in CI. It now takes ~10s, and block/buzz (5,873 files) and openai/codex (9,212 files) both complete in seconds. Output is byte-identical before and after.

Only check what changed

npx agenttruth --changed origin/main

Scans only the doc files a branch touched, so the check is cheap enough to run on every push instead of nightly. Falls back to a full scan when the ref is missing or the tree isn't a git repo.

CI

- uses: hemv-857/agenttruth@v0

Or run it directly:

- run: npx agenttruth

Adopting without breaking your build

A repo that has been drifting for a year cannot adopt a checker that fails on day one — nobody reads 40 findings, everyone uninstalls. Baseline the existing findings, then CI fails only on new drift, which is the only thing anyone can act on:

npx agenttruth --update-baseline   # commit .agenttruth-baseline.json

Fix what you can, then re-baseline. Fingerprints ignore line numbers, so unrelated edits won't resurrect a suppressed finding.

Config (optional)

agenttruth.config.json — every field has a working default:

{
  "docs": ["README.md", "CLAUDE.md", "AGENTS.md"],
  "docsDir": "docs",
  "agentDocs": ["CLAUDE.md", "AGENTS.md", "GEMINI.md"],
  "checks": ["numeric-claim", "missing-file", "broken-link", "missing-command"],
  "ignore": ["CHANGELOG.md"],
  "driftTolerance": 0.25
}

Files in agentDocs are discovered at any depth, because that is how agents load them — Claude Code walks up from the directory it is working in, Codex reads AGENTS.md in the folder being edited, so a stale src/features/agents/AGENTS.md misleads exactly like a stale root one. Every other doc is read from the root and docs/ only: recursing for README.md would pick up a README per package.

MCP

Agents can check their own instructions before a run, and fix them when stale:

{ "mcpServers": { "agenttruth": { "command": "npx", "args": ["-y", "agenttruth-mcp"] } } }

Zero dependencies — it speaks JSON-RPC over stdio directly.

GitHub Action

- run: npm install --no-save agenttruth
- uses: hemv-857/agenttruth@v0
  with:
    fail-on: error
    changed: origin/main     # only docs this branch touched

Findings annotate the pull request at the exact file and line, next to the diff that caused them — no code-scanning setup required. fail-on defaults to error so the action blocks on provable findings without failing builds on judgement calls.

The v0 tag moves with each release, so you get fixes without editing the workflow. Pin @v0.8.0 instead if you want the version frozen.

Prefer code scanning annotations? sarif: agenttruth.sarif writes a SARIF report for github/codeql-action/upload-sarif, and the action then skips its own annotations so nothing appears twice.

Options

--dir <path>          repo root (default: cwd)
--json                machine-readable output (--format json)
--format <fmt>        text | json | github
--fail-on <sev>       error | warning | any | none (default: any) — exit threshold
--update-baseline     record current findings as known
--fix                 rewrite unambiguous stale counts to the measured value
--sarif [file]        write SARIF 2.1.0 for code scanning
--changed <ref>       only check docs changed since this git ref
-q, --quiet           print only when there are findings

--format github prints workflow commands, so plain - run: npx agenttruth annotates a PR diff too. It needs no code scanning, which matters on private repos where that requires GitHub Advanced Security.

Honest scope

Four checks (numeric-claim, missing-file, broken-link, missing-command), each tuned for low false positives over recall. Errors and warnings are separated so --fail-on error can block on the provable ones and leave judgement calls visible. If it can't prove a file is wrong, it stays quiet.

Measured on sixteen repos — claude-code, pretext, openai/codex, agent-browser, block/buzz, express, vite, eslint, vitest, babel, and six Python/Go/Rust projects (requests, flask, fastapi, gin, cobra, ripgrep) — it reports 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0. Every finding was inspected line by line. Earlier builds reported eight times as much here, and most of it was wrong: files belonging to a sibling repo, the reader's home directory, another language's source, a compiler header, or an HTTP route. Those classes are now decided by ownership rather than pattern — a claim about this tree must cite this tree. The three surviving hits are real: babel's CONTRIBUTING.md says pluginsConfig.json where the repo ships pluginConfig.json, agent-browser links a docs page that moved, and buzz lists an e2e_mcp.rs test file that does not exist.

It behaves identically on a macOS checkout and a case-sensitive Linux CI runner. Docs are matched against their real on-disk names, so a repo shipping Readme.md is checked everywhere, and annotations point at a path that exists.

License

MIT