ProofBrain

A company brain for AI agents, with proof the output is good.

Click the diagram to open it full size.

Shared memory your people and agents both write to, and numbers that show whether the AI output is any good.

ProofBrain is an MCP server that gives agents (Claude Code, Cursor, your own) one governed place to read and write company knowledge:

1. Shared memory, the company brain. Docs, code and notes are indexed for hybrid search. On top of that sits a git-backed memory that both people and agents write to. Agents propose facts; a different person approves them. Every entry has an author, a reviewer and a history.

2. Measurability. Every answer, email draft and learned rule is written to an ai_events log, together with its sources, cost and the human verdict. Quality numbers come from that log, not from impressions:

   • retrieval recall and MRR;

   • the share of answer claims backed by their citation;

   • how much humans edit drafts, reported as a distribution;

   • whether a type of task has earned autonomy, and what went out without review.

The repo ships a synthetic demo company, Cobblestone Cycles, an e-bike subscription business: 22 docs, a small Python service, 7 memory entries, 340 customer emails and 56 evaluation questions. docker compose up followed by proofbrain demo (email loop) and proofbrain eval (retrieval and answers) reproduces every number below; proofbrain report writes them to one file.

Measuring what the AI produces

Numbers below come from the offline, deterministic baselines that ship with the repo: a hashing embedder, an extractive answerer and a template drafter. They show what the harness measures, not a ceiling. The retrieval and answer tables run against a real model by setting PROOFBRAIN_EMBEDDER / PROOFBRAIN_LLM (litellm:<model>). The email table cannot: its simulated reviewer edits template drafts (see Limits).

Phase 1: the email loop learns from edits. Four steps:

1. 24 customer emails are drafted.

2. A reviewer edits each draft before sending.

3. ProofBrain compacts the 24 (draft, sent) pairs into style rules, and the reviewer approves them into shared memory.

4. 16 held-out emails are drafted twice, without and with the rules, and compared with what the reviewer sent.

Edit distance is word-level Levenshtein divided by the longer text: 0 means sent as drafted, 1 means rewritten. The rules learn the house style; what remains is the personal touches the reviewer adds case by case, which no rule should learn.

Phase 2: drafts go out on their own, one type of email at a time. Each type of email (failed payment, damage report, ...) has its own autonomy gate. It opens when at least 80% of the last 20 human-reviewed drafts of that type, written with the current rules, were sent unchanged. While it is open, drafts are sent without review, except every 5th, which still goes to a human. Those spot checks keep the window fresh: if the unchanged rate falls below 80%, the gate closes and every draft goes back to review. A new version of the rules, or another drafting model, starts the gate from zero. 300 more emails arrive after the rules are approved:

The gates open where the simulated reviewer rarely adds anything: routine notices such as a failed card payment. That is how the simulation is set up, so this table shows that the gate opens and spot-checks as designed, not that the drafts are good enough to send unreviewed (Limits). No gate closes again in this run; re-closing is covered by the tests (tests/test_autonomy.py).

Retrieval (51 answerable questions, document level):

What the ablations say:

• Shared memory carries knowledge the docs don't have. Without it, recall on the memory questions drops from 0.75 to 0.

• Trigram matching hurts plain-language questions. It now runs only when a query contains an identifier (swap_scheduler, pricing.py).

• The offline embedder adds little on top of full-text search, as expected for a lexical embedder. Measuring real embeddings is one environment variable away.

Answers (56 questions, including 5 with no answer in the corpus):

Correct means every expected key phrase appears in the answer. A claim is grounded when the source it cites contains most of the claim's words. The extractive baseline answers only half the questions: it cannot paraphrase or combine sources, and it never declines. Those are the first things an LLM answerer has to improve, and this table is where the improvement would show.

The full report, with per-category recall and the edit-distance histogram, is make report → reports/report.md. Limits of this evaluation explains what these numbers do not show.

Related MCP server: io.github.vouchdev/vouch

Quickstart

docker compose up -d --build                       # PostgreSQL + pgvector, MCP server on :8790
docker compose exec app proofbrain demo            # index Cobblestone Cycles, run the email loop
docker compose exec app proofbrain eval            # retrieval + answers with ablations
docker compose exec app proofbrain report          # reports/report.md inside the container
docker compose exec app proofbrain ask "Can support give a EUR 50 goodwill credit?"

A support agent may give a goodwill credit up to EUR 40 without approval. [1] Anything above EUR 40 needs a team lead. [1]

  [1] company:docs/policies/refunds.md#L34-L37  Cancellation and Refund Policy

  grounded claims: 2/2   event #582

Connect an agent

docker compose exec app proofbrain token create my-agent --roles reader,writer,email   # printed once
claude mcp add --transport http proofbrain http://localhost:8790/mcp \
  --header "Authorization: Bearer pb_..."

For a local stdio server, run proofbrain serve (stdio is the default transport). The identity comes from PROOFBRAIN_LOCAL_PRINCIPAL and PROOFBRAIN_LOCAL_ROLES. examples/agent/session.py is a scripted agent session over HTTP.

Without Docker

uv sync
make test-db-up                                    # throwaway pgvector on :55441
export PROOFBRAIN_DSN=postgresql://proofbrain:proofbrain@127.0.0.1:55441/proofbrain
uv run proofbrain demo --reset && uv run proofbrain eval && uv run proofbrain report
make test lint

MCP tools

An agent sees only the tools its token's roles allow: tools/list is filtered, tools/call is checked against the same policy, and every call is audited.

Architecture: the company brain

The diagram at the top shows how the pieces work together. The code is split into four parts:

• packages/core: storage, hybrid search, shared memory, the email loop, the event log and metrics. Every interface (MCP, CLI, eval) goes through one Brain facade, so the eval measures the same code path agents use.

• packages/indexer:

  • connectors (a local folder for now);

  • markdown and python chunkers that keep line numbers for citations;

  • content-hash change detection, so a re-run touches only changed files;

  • secret scrubbing before anything is embedded.

• packages/mcp: the MCP server (mcp SDK 2.x), the access middleware, streamable HTTP with a bearer gate and /healthz.

• src/proofbrain: the CLI, the demo loader, the evaluation and the report.

Design decisions and their trade-offs are in docs/adr/.

Security model

SECURITY.md has the full model:

• Read-only by design. The only writes are memory proposals and email drafts. Memory proposals always wait for a human; drafts do too, unless the autonomy gate for their type of email is open. ProofBrain has no mail transport: "sent" is a thread status and a verdict in the event log.

• Tokens are stored hashed.

• Per-role tool visibility. An agent sees only the tools its roles allow.

• Every call is audited, allowed and denied.

• Known secret formats stay out of the index, memory and logs. Tokens and keys with a recognizable shape (cloud keys, API tokens, bearer headers, KEY=value assignments, URL passwords) are scrubbed before indexing and before audit and ai_events rows are written, and rejected by memory. Opaque strings without such a shape are not detected, except as values of secret-named fields.

• CI runs gitleaks over the full history, plus semgrep and pip-audit.

Limits

• Synthetic data, small set. The data is invented and the question set has 56 items. A few points of difference is noise.

• Offline baselines by default. The numbers above come from deterministic stand-ins, not from an LLM or a semantic embedder. The LiteLLM paths (PROOFBRAIN_LLM, PROOFBRAIN_EMBEDDER) share the same interfaces and are covered by the same pipeline. Their numbers are not in this README because no model was called to produce it.

• Tuning used this question set. The answer threshold was chosen on odd-numbered questions and checked on even ones; both score about 0.50. A larger held-out set is needed before trusting small gains.

• The simulated reviewer is consistent; people are not. Expect smaller gains from learned rules with real reviewers. The reviewer's "sent" replies are edits of the template drafter's output, so the email table measures the learning loop, not draft quality, and an LLM drafter would need real reviewers to be compared.

• The autonomy gate demonstrates the mechanism, not draft quality. How often the simulated reviewer adds a personal touch is set by hand per type of email (TOUCH_RATE in src/proofbrain/demo/mailgen.py): rarely for routine notices, often for upset customers. Those rates decide which gates open, and the reviewer does exactly what it is programmed to do. Nothing has been tried with real reviewers, so an open gate here is not evidence that drafts can be sent without review.

Development

make help        # sync, lint, fmt, test-db-up, test, demo, eval, report, audit

Stack: Python 3.12+, uv workspace, PostgreSQL 18 with pgvector and pg_trgm, psycopg 3, the MCP Python SDK 2.x, LiteLLM (optional).

License

Apache-2.0. Copyright 2026 Wiaczesław Cyunczyk.