Riposte

Parry the claim. Answer with proof.

AI agents decide, act, and report "done." Riposte checks those things against the documents and the real state — deterministically, with no model in the loop — and answers PASS, FAIL, or ABSTAIN (it can't verify, so a person decides). Every answer carries a replayable proof, and every result can be written to a hash-chained, signed ledger that anyone can verify without trusting us.

It is independent of the agent, the model, and the model vendor: the same input gives the same verdict, byte for byte.

Measured on someone else's benchmark

Distil Labs published an accounts-payable benchmark (Apache-2.0): invoices that arrive as free-text emails, checked against a purchase order and a goods receipt. Their conclusion: "That is why this is a job for a model and not for a short function." We tested the short function.

The full write-up is packages/verify/docs/BENCHMARK-AP.md. Every number re-runs from bench/ap.

Related MCP server: Mcp-Omega-Brain

Rules that write themselves

Someone has to write the rules. Unless the rules can be learned. From the same 100 invoices, labelled only paid or held, verify mine rediscovered all 4 of the policy's rules, with 0 false rules. That includes the 2% price tolerance, learned from the labels with the interval the data supports. The 4 rules explain all 68 holds, and every rejected candidate is recorded with its cause of death. The rules compile into an ordinary ruleset: enforcing them, verify agrees with the gold decision on 100 of 100 invoices. The method was pre-registered before the miner was written. Write-up: BENCHMARK-MINE.md. That result is on synthetic data. The first real-data tests are below, and they did not all pass.

Honest scope: the benchmark is synthetic and templated, and real invoices will abstain more often. Abstaining is the designed way to fail, because a person looks at it. Coverage is always published next to accuracy.

On real SEC filings: including the misses

Three studies on every 10-K in two quarters of the SEC's public Financial Statement Data Sets. Each was pre-registered before its data was downloaded, and each is published whether it passed or not (BENCHMARK-SEC.md, harness in bench/sec).

What the misses taught us:

• In 166 audited flags there was no genuine accounting error. Every correct flag was a number filed under the wrong XBRL element.

• Filed statements are among the most-checked numbers anywhere. These identities earn their keep checking an AI's extraction of a statement, where the errors are introduced. That is the next pre-registered test.

What's here

Install in Claude Code (one plugin)

claude plugin marketplace add SorenSocordia/riposte
claude plugin install riposte@riposte            # -s project or -s local to scope it to one project

It adds a Stop hook that checks the agent's closing claims against its own tool results, the MCP tools, and a receipt ledger. It is record-only by default: it measures and never forces a turn until you set RIPOSTE_HOOK_MODE. See plugins/riposte.

Quick start (from source; not yet on npm)

git clone https://github.com/SorenSocordia/riposte.git && cd riposte
npm install
npm run build
npm test

Check an invoice against its purchase order and goods receipt:

node packages/verify/dist/cli.js ap packages/verify/docs/examples/ap-three-way.txt
# → hold_quantity: "Machine grease cartridge 14 oz" billed 10, received 7 — with the exact lines it read. Exit 1 (hold).

Give an agent receipts (any MCP host):

{ "mcpServers": { "riposte": {
    "command": "node", "args": ["<path>/riposte/packages/receipts/dist/stdio.js"],
    "env": { "RECEIPTS_ROOT": "<the agent's workspace>" } } } }

Re-run the benchmark:

npx tsx bench/ap/bench.ts --run 1              # the blind run — checker sha256 d1ad333f…, 82 decided, 0 wrong
npx tsx bench/ap/audit.ts                      # gate every published model decision
npx tsx bench/ap/stress.ts --seed holdout1     # … through holdout5

Principles

• Deterministic. The same input, ruleset and engine version always produce the same verdict id, byte for byte.

• Abstain, never guess. A value that can't be read unambiguously comes back as can't tell, with the reason.

• Independent. No model calls and no network at runtime. It runs on-prem or air-gapped.

• Published error. Accuracy is reported alongside coverage, and blind runs are kept separate from tuned ones.

License

Apache-2.0. Third-party data and its licences are listed in NOTICE.