# retained

**An MCP memory server whose writes are provable.**

A memory server answers `ok` and you move on. That answer is a promise about bytes, and almost nobody
tests it. `retained` makes the promise checkable: every write is appended, fsynced, read back from the
file and hash-chained to its predecessor **before** it is acknowledged; every read says how much it
did *not* return; and the search index can be reconciled against the log it came from, in both
directions.

```
$ python retained.py --selftest
retained selftest: 14 ok, 0 failed
```

## Three invariants

| | invariant | what it means |
|---|---|---|
| **I1** | no acknowledgement without proof | append → fsync → byte-exact read-back → chain check; the reply carries `durable` and the four checks |
| **I2** | no silent loss on read | every search returns `total_matched`, `returned`, `truncated` |
| **I3** | immutable records, explicit supersede | a correction is a new record; the superseded one is marked, never overwritten |

The store is the truth: an append-only JSONL log, one hash-chained record per line. The search index
is a SQLite FTS5 table that can be deleted and rebuilt at any time. `memory_verify` compares them and
reports drift **both ways**, because "the index is ahead of the store" is how a memory system loses
things while reporting health.

## Tools

| tool | arguments | returns |
|---|---|---|
| `memory_write` | `text`, `kind?`, `scope?`, `source?`, `supersedes?`, `client_op_id?`, `fields?` | `id`, `seq`, `sha256`, `durable`, `checks{append,fsync,readback_match,chain}`, `warnings` |
| `memory_search` | `query`, `scope?`, `k?`, `max_tokens?` | `hits`, `total_matched`, `returned`, `truncated` |
| `memory_get` | `ids?`, `limit?` | full records, `missing`, `corrupt_lines`, `chain_ok` |
| `memory_verify` | none | `records`, `chain_ok`, `corrupt_lines`, `missing_from_index`, `index_only`, `coverage`, `intact` |
| `memory_checkpoint` | `label`, `summary?`, `scope?` | a checkpoint record, same proof as a write |

Unknown fields are stored verbatim at the top level of the record. Nothing is ever rewritten, so a
field this server does not model cannot be lost by it.

## Measured against the same probes as everybody else

[memory-conformance](https://github.com/arm00pv/memory-conformance) runs six falsifiable retention
probes against a server over MCP stdio. Results:

| subject | result |
|---|---|
| `@modelcontextprotocol/server-memory@1.0.0` | 5/6 — fails `foreign_field` (a field written from outside is silently dropped) |
| a minimal control server | 6/6 |
| **retained** | **6/6** — including 50/50 concurrent writes across two server processes |

```
$ python harness.py --adapter=retained --cmd="python /path/to/retained.py"
ack_readback     PASS   restart        PASS   foreign_field  PASS
foreign_record   PASS   concurrent     PASS   large_record   PASS
0 probe(s) failed
```

## Hooks: the difference between a memory and a library

An MCP server **cannot initiate a request** - that is in the protocol, not in this implementation. A
store reachable only through a tool call is memory the model has to remember to ask for, which means
it usually will not. The fix is a host hook that injects a brief at session start:

```
$ RETAINED_DIR=~/.retained python retained.py --brief --budget=120
memory: 3 record(s)
- [checkpoint] shipped retained 0.2.0 with optional embeddings; retrieval 11/12 h1, 12/12 h3
- [note] conformance: reference 5/6 retention, 0/12 retrieval
- [note] silentfail issue 1: dead-guard is quadratic, 58s of a 134s scan
```

Newest first, checkpoints before notes, and every brief says how much it left out - a brief that
silently truncates would be this project's own defect one level up. `--json` gives the same content
with `included`, `dropped` and `tokens` for a host that wants to budget.

Claude Code (`.claude/settings.json`):

```json
{ "hooks": { "SessionStart": [ { "hooks": [ { "type": "command",
  "command": "RETAINED_DIR=$HOME/.retained python3 -m retained --brief --budget=800" } ] } ] } }
}
```

`hooks/session-start.sh` is a portable version for any harness that can run a command per session.
Without a hook, `retained` is still a correct store - it is just a store nobody consults.

## Retrieval: lexical, and optionally semantic

By default the index is SQLite FTS5 and nothing else. Set one environment variable and it becomes a
hybrid:

```
RETAINED_EMBED_URL=http://127.0.0.1:11434/api/embeddings   # any Ollama-style endpoint
RETAINED_EMBED_MODEL=all-minilm                           # 384 dimensions, ~46 MB
```

Vectors are stored in the index (never in the log - the log is the truth and stays dependency-free),
and search fuses the lexical and vector rankings with reciprocal rank fusion. If the embedder is down,
writes still land and search still answers lexically: an optional feature must not become a single
point of failure.

Measured on the twelve-question retrieval eval in
[memory-conformance](https://github.com/arm00pv/memory-conformance) - six questions sharing vocabulary
with the memory they want, six paraphrases:

| mode | hit@1 | hit@3 |
|---|---|---|
| lexical only | 10/12 | 11/12 |
| hybrid (all-minilm) | **11/12** | **12/12** |

The one lexical miss - *"how many machines do we run"* against *"three hosts: throne, watchtower and
pi"* - is found at rank 3 with vectors and still not at rank 1, which is a fair picture of a 384-
dimension model on a twelve-item corpus: it closes the gap, it does not erase it.

## The bug the selftest caught on its first run

The read-back check failed immediately: the file was 183 bytes where 182 were written. Without
`O_BINARY`, Windows opens the file in text mode and turns every LF into CRLF, so the bytes on disk are
not the bytes written. One flag, found in the first minute - and it is the same carriage-return family
that has broken appends in the estate this came from twice already.

## Install and run

```
pip install git+https://github.com/arm00pv/retained
RETAINED_DIR=~/.retained retained        # MCP server on stdio
```

Register it in any MCP client as a stdio server:

```json
{ "mcpServers": { "retained": { "command": "retained", "env": { "RETAINED_DIR": "~/.retained" } } } }
}
```

## Limits, stated

- **Retrieval is lexical.** SQLite FTS5 with `bm25` ranking. No embeddings, no semantic search; a
  memory that needs "find the thing I meant" needs more than this.
- **One writer at a time.** The append takes an OS lock (`flock` / `msvcrt.locking`); concurrent
  processes serialise rather than merge.
- **No hooks.** An MCP server cannot initiate a request, so injection has to be driven by the client.
  Without a hook in your harness, this is a tool the model must remember to call.
- **No encryption, no auth, no multi-user scopes beyond the `scope` field.**
- **Fourteen self-test checks and six conformance probes are not a proof of correctness.** They are
  the failures that were thought of, and the list above is the rest.

MIT licensed.