# ProofBrain

**A company brain for AI agents, with proof the output is good.**

<a href="docs/architecture.svg?raw=true"><img src="docs/architecture.svg" width="100%" alt="How ProofBrain works: agents learn from human edits under review, pass a gate when 80% of recent replies are sent unchanged, then answer on their own with spot checks; every draft and its outcome is logged"></a>

<sub>Click the diagram to open it full size.</sub>

Shared memory your people and agents both write to, and numbers that show whether the AI output is any good.

ProofBrain is an [MCP](https://modelcontextprotocol.io) server that gives agents (Claude Code, Cursor, your own)
one governed place to read and write company knowledge:

1. **Shared memory, the company brain.** Docs, code and notes are indexed for hybrid search. On top of
   that sits a git-backed memory that both people and agents write to. Agents *propose* facts; a different
   person *approves* them. Every entry has an author, a reviewer and a history.
2. **Measurability.** Every answer, email draft and learned rule is written to an `ai_events` log, together with
   its sources, cost and the human verdict. Quality numbers come from that log, not from impressions:
   - retrieval recall and MRR;
   - the share of answer claims backed by their citation;
   - how much humans edit drafts, reported as a distribution;
   - whether a type of task has earned autonomy, and what went out without review.

The repo ships a synthetic demo company, **Cobblestone Cycles**, an e-bike subscription business: 22 docs, a
small Python service, 7 memory entries, 340 customer emails and 56 evaluation questions. `docker compose up`
followed by `proofbrain demo` (email loop) and `proofbrain eval` (retrieval and answers) reproduces every
number below; `proofbrain report` writes them to one file.

## Measuring what the AI produces

Numbers below come from the offline, deterministic baselines that ship with the repo: a hashing embedder, an
extractive answerer and a template drafter. They show what the harness measures, not a ceiling. The retrieval and
answer tables run against a real model by setting `PROOFBRAIN_EMBEDDER` / `PROOFBRAIN_LLM` (`litellm:<model>`).
The email table cannot: its simulated reviewer edits template drafts (see [Limits](#limits)).

**Phase 1: the email loop learns from edits.** Four steps:
1. 24 customer emails are drafted.
2. A reviewer edits each draft before sending.
3. ProofBrain compacts the 24 (draft, sent) pairs into style rules, and the reviewer approves them into shared
   memory.
4. 16 *held-out* emails are drafted twice, without and with the rules, and compared with what the reviewer sent.

| held-out drafts (n=16) | sent unchanged | edit distance mean | p50 | p90 |
|---|---|---|---|---|
| without learned rules | 0% | 0.54 | 0.61 | 0.63 |
| with learned rules | 56% | 0.06 | 0.00 | 0.19 |

Edit distance is word-level Levenshtein divided by the longer text: 0 means sent as drafted, 1 means rewritten.
The rules learn the house style; what remains is the personal touches the reviewer adds case by case, which no
rule should learn.

**Phase 2: drafts go out on their own, one type of email at a time.** Each type of email (failed payment, damage
report, ...) has its own autonomy gate. It opens when at least 80% of the last 20 human-reviewed drafts of that
type, written with the current rules, were sent unchanged. While it is open, drafts are sent without review,
except every 5th, which still goes to a human. Those spot checks keep the window fresh: if the unchanged rate
falls below 80%, the gate closes and every draft goes back to review. A new version of the rules, or another
drafting model, starts the gate from zero.
300 more emails arrive after the rules are approved:

| type of email | drafts | reviewed by a human | sent without review | spot checks unchanged | gate |
|---|---|---|---|---|---|
| failed payment | 35 | 19 | 16 | 3 of 3 | **open** from draft 17, 95% unchanged |
| late cancellation | 57 | 26 | 31 | 6 of 7 | **open** from draft 20, 95% unchanged |
| late swap | 53 | 53 | 0 | – | closed, 65% unchanged |
| damage report | 54 | 54 | 0 | – | closed, 60% unchanged |
| delivery delay | 40 | 40 | 0 | – | closed, 55% unchanged |
| early cancellation | 61 | 61 | 0 | – | closed, 55% unchanged |

The gates open where the simulated reviewer rarely adds anything: routine notices such as a failed card payment.
That is how the simulation is set up, so this table shows that the gate opens and spot-checks as designed, not
that the drafts are good enough to send unreviewed ([Limits](#limits)). No gate closes again in this run; re-closing
is covered by the tests (`tests/test_autonomy.py`).

**Retrieval** (51 answerable questions, document level):

| variant | recall@5 | MRR@10 |
|---|---|---|
| hybrid (vector + full-text, RRF) | 0.91 | 0.74 |
| hybrid + trigram on every query | 0.91 | 0.64 |
| vector only (hashing embedder) | 0.83 | 0.62 |
| full-text only | 0.91 | 0.74 |
| hybrid without shared memory | 0.84 | 0.68 |

What the ablations say:
- **Shared memory carries knowledge the docs don't have.** Without it, recall on the memory questions drops from
  0.75 to 0.
- **Trigram matching hurts plain-language questions.** It now runs only when a query contains an identifier
  (`swap_scheduler`, `pricing.py`).
- **The offline embedder adds little on top of full-text search, as expected for a lexical embedder.** Measuring
  real embeddings is one environment variable away.

**Answers** (56 questions, including 5 with no answer in the corpus):

| variant | correct | grounded claims | declined when unanswerable |
|---|---|---|---|
| full (hybrid + memory) | 0.55 | 1.00 | 0.00 |
| without shared memory | 0.49 | 1.00 | 0.00 |
| without retrieval | 0.00 | – | 1.00 |

*Correct* means every expected key phrase appears in the answer. A claim is *grounded* when the source it cites
contains most of the claim's words. The extractive baseline answers only half the questions: it cannot
paraphrase or combine sources, and it never declines. Those are the first things an LLM answerer has to improve,
and this table is where the improvement would show.

The full report, with per-category recall and the edit-distance histogram, is `make report` →
`reports/report.md`. [Limits of this evaluation](#limits) explains what these numbers do not show.

## Quickstart

```bash
docker compose up -d --build                       # PostgreSQL + pgvector, MCP server on :8790
docker compose exec app proofbrain demo            # index Cobblestone Cycles, run the email loop
docker compose exec app proofbrain eval            # retrieval + answers with ablations
docker compose exec app proofbrain report          # reports/report.md inside the container
docker compose exec app proofbrain ask "Can support give a EUR 50 goodwill credit?"
```

```
A support agent may give a goodwill credit up to EUR 40 without approval. [1] Anything above EUR 40 needs a team lead. [1]

  [1] company:docs/policies/refunds.md#L34-L37  Cancellation and Refund Policy

  grounded claims: 2/2   event #582
```

### Connect an agent

```bash
docker compose exec app proofbrain token create my-agent --roles reader,writer,email   # printed once
claude mcp add --transport http proofbrain http://localhost:8790/mcp \
  --header "Authorization: Bearer pb_..."
```

For a local stdio server, run `proofbrain serve` (stdio is the default transport). The identity comes from
`PROOFBRAIN_LOCAL_PRINCIPAL` and `PROOFBRAIN_LOCAL_ROLES`. [`examples/agent/session.py`](examples/agent/session.py)
is a scripted agent session over HTTP.

### Without Docker

```bash
uv sync
make test-db-up                                    # throwaway pgvector on :55441
export PROOFBRAIN_DSN=postgresql://proofbrain:proofbrain@127.0.0.1:55441/proofbrain
uv run proofbrain demo --reset && uv run proofbrain eval && uv run proofbrain report
make test lint
```

## MCP tools

An agent sees only the tools its token's roles allow: `tools/list` is filtered, `tools/call` is checked against
the same policy, and every call is audited.

| family | tool | role | what it does |
|---|---|---|---|
| search | `search` | reader | Hybrid search with citations (`source:path#Lstart-Lend`) and per-retriever ranks |
| | `ask` | reader | Answer with per-claim citations and a grounding check; logged as an AI event |
| | `get_document`, `list_sources` | reader | Full document with line numbers; indexed sources |
| memory | `memory_list`, `memory_read` | reader | Company brain entries, with who proposed and who approved each change |
| | `memory_save`, `memory_delete` | writer | Propose a change: nothing lands until a reviewer approves; secrets are rejected |
| | `memory_pending`, `memory_review` | reviewer | Four-eyes review: the proposer cannot approve their own change |
| email | `email_inbox`, `email_thread` | email | Customer threads and the facts our systems hold about the case |
| | `email_draft` | email | Draft using policy sources and the approved style rules; sent without review if the gate for its type is open |
| | `email_feedback` | email | What was actually sent: the edit distance becomes the quality signal and feeds the gate |
| | `email_rules` | email | Show the rules, or compact reviewed pairs into a rules proposal |
| metrics | `brain_metrics` | reader | Grounded-claim rate, abstentions, edit-distance distribution, cost, latency, one autonomy gate per type of email |
| access | `access_summary` | reader | Who am I, which roles, which tools |
| admin | `audit_tail` | admin | Recent tool calls, allowed and denied |

## Architecture: the company brain

The [diagram at the top](docs/architecture.svg?raw=true) shows how the pieces work together. The code is split into four parts:

- **`packages/core`**: storage, hybrid search, shared memory, the email loop, the event log and metrics. Every
  interface (MCP, CLI, eval) goes through one `Brain` facade, so the eval measures the same code path agents use.
- **`packages/indexer`**:
  - connectors (a local folder for now);
  - markdown and python chunkers that keep line numbers for citations;
  - content-hash change detection, so a re-run touches only changed files;
  - secret scrubbing before anything is embedded.
- **`packages/mcp`**: the MCP server (`mcp` SDK 2.x), the access middleware, streamable HTTP with a bearer gate
  and `/healthz`.
- **`src/proofbrain`**: the CLI, the demo loader, the evaluation and the report.

Design decisions and their trade-offs are in [`docs/adr/`](docs/adr).

## Security model

[SECURITY.md](SECURITY.md) has the full model:
- **Read-only by design.** The only writes are memory proposals and email drafts. Memory proposals always wait
  for a human; drafts do too, unless the autonomy gate for their type of email is open. ProofBrain has no mail
  transport: "sent" is a thread status and a verdict in the event log.
- **Tokens are stored hashed.**
- **Per-role tool visibility.** An agent sees only the tools its roles allow.
- **Every call is audited**, allowed and denied.
- **Known secret formats stay out of the index, memory and logs.** Tokens and keys with a recognizable shape
  (cloud keys, API tokens, bearer headers, `KEY=value` assignments, URL passwords) are scrubbed before
  indexing and before audit and `ai_events` rows are written, and rejected by memory. Opaque strings without
  such a shape are not detected, except as values of secret-named fields.
- **CI** runs gitleaks over the full history, plus semgrep and pip-audit.

## Limits

- **Synthetic data, small set.** The data is invented and the question set has 56 items. A few points of
  difference is noise.
- **Offline baselines by default.** The numbers above come from deterministic stand-ins, not from an LLM or a
  semantic embedder. The LiteLLM paths (`PROOFBRAIN_LLM`, `PROOFBRAIN_EMBEDDER`) share the same interfaces and
  are covered by the same pipeline. Their numbers are not in this README because no model was called to produce
  it.
- **Tuning used this question set.** The answer threshold was chosen on odd-numbered questions and checked on
  even ones; both score about 0.50. A larger held-out set is needed before trusting small gains.
- **The simulated reviewer is consistent; people are not.** Expect smaller gains from learned rules with real
  reviewers. The reviewer's "sent" replies are edits of the template drafter's output, so the email table
  measures the learning loop, not draft quality, and an LLM drafter would need real reviewers to be compared.
- **The autonomy gate demonstrates the mechanism, not draft quality.** How often the simulated reviewer adds a
  personal touch is set by hand per type of email (`TOUCH_RATE` in `src/proofbrain/demo/mailgen.py`): rarely for
  routine notices, often for upset customers. Those rates decide which gates open, and the reviewer does exactly
  what it is programmed to do. Nothing has been tried with real reviewers, so an open gate here is not evidence
  that drafts can be sent without review.

## Development

```bash
make help        # sync, lint, fmt, test-db-up, test, demo, eval, report, audit
```

Stack: Python 3.12+, uv workspace, PostgreSQL 18 with pgvector and pg_trgm, psycopg 3, the MCP Python SDK 2.x,
LiteLLM (optional).

## License

Apache-2.0. Copyright 2026 Wiaczesław Cyunczyk.