# mcp-zeromem

Conversational memory for agents that costs zero tokens to maintain, over MCP.

Every turn an agent sees goes into one SQLite store. Recall over that store —
entity graph, lexical and dense search — runs with no LLM in the loop, so
remembering is free and the only tokens spent are the ones the model spends
reading what came back. The method follows the Zero-Mem paper,
[arXiv:2607.29377](https://arxiv.org/abs/2607.29377), whose upstream
implementation is [ptaranat/zeromem](https://github.com/ptaranat/zeromem); the
implementation here is a clean-room rewrite in Rust, loaded in-process by a
Node MCP server, with persistent indexes so opening a large store does not mean
rebuilding it.

**Status: Parts 1 and 2 complete.** The engine stores turns with their
entity spans, tokens and embeddings in one transaction, keeps the entity
graph, temporal hierarchy and lexical statistics on disk stamped with a
generation counter, and recalls over four views (lexical, entity, dense,
recent) fused by reciprocal rank and calibrated into a primary/supporting
evidence set. The Node server exposes six MCP tools over Streamable HTTP and
stdio plus a REST API. The admin UI shows the store's structure, not just its
counts: an entity graph, a timeline of the temporal hierarchy, a session
inspector with entity spans, a retrieval trace for any query, an embedding
map, a side-by-side compare, growth and health charts, and the eval history.
Quality is gated by a labeled eval set; a 50k-turn store opens in under
300 ms.

## Quick start

```bash
cp .env.example .env
# Fill in MCP_ZEROMEM_TOKEN:  openssl rand -hex 32
docker compose up -d --build
curl -s localhost:3200/api/status | jq .
```

Then point a client at `http://localhost:3200/mcp` with
`Authorization: Bearer $MCP_ZEROMEM_TOKEN`. For Claude Code:

```bash
claude mcp add --transport http zeromem http://localhost:3200/mcp \
  --header "Authorization: Bearer $MCP_ZEROMEM_TOKEN"
```

The admin UI is at `http://localhost:3200/`; it asks for the same token.

Locally, without Docker (needs a Rust toolchain and Node 26):

```bash
npm install
npm run build:native      # builds the addon; rerun after any Rust change
SECURE_LOCAL_NET=true npm run dev
```

## Writing to the store

Three ways in, all landing in the same `./data/zeromem.db`:

- **MCP write tools** (`zeromem_remember`, `zeromem_ingest`), for an agent to
  store what it decides is worth keeping.
- **`zm` inside the container**, for hooks and bulk loads with no server hop:

  ```bash
  printf '{"session_id":"s1","speaker":"user","text":"we chose postgres"}\n' \
    | docker compose exec -T mcp-zeromem zm ingest
  ```

- **`zm` on the host**, pointed at the bind-mounted directory
  (`ZEROMEM_HOME=./data zm ingest`). The store is SQLite in WAL mode, so a host
  writer and the server coexist; the server notices new turns on its next read.

Turns are deduplicated by content (or by an explicit `uuid`), so re-ingesting a
transcript is idempotent and reports `duplicates` rather than double-counting.

### Remembering Claude Code sessions automatically

`zm hook` reads a Claude Code hook event (`{"transcript_path", "session_id"}`)
from stdin and remembers the transcript's user and assistant turns. Wire it to
`Stop` and `SessionEnd` in `~/.claude/settings.json`. Two ways to run it:

- **Host `zm`** (simplest; needs `cargo install --path crates/zeromem-core`):

  ```json
  { "hooks": { "Stop": [{ "hooks": [{ "type": "command",
      "command": "ZEROMEM_HOME=$HOME/mcp-zeromem/data zm --embedder none hook" }] }] } }
  ```

  `--embedder none` keeps the hook from loading a model or calling an
  endpoint: the turns land without vectors and the server embeds them with
  the store's embedder on its next read (they are searchable lexically in the
  meantime). Leave the flag off to have the hook embed inline with whatever
  the store uses, which needs `ZEROMEM_MODELS` pointing at `data/models` for
  the ONNX model — or run it in the container:

- **`zm` in the container**, with the transcripts mounted read-only and the
  path prefix mapped:

  ```yaml
  # docker-compose.override.yml
  services:
    mcp-zeromem:
      volumes:
        - ~/.claude:/host-claude:ro
      environment:
        ZEROMEM_HOOK_MAP: /home/me/.claude=/host-claude
  ```

  ```json
  { "type": "command", "command": "docker compose -f ~/mcp-zeromem/docker-compose.yml exec -T mcp-zeromem zm hook" }
  ```

  `ZEROMEM_HOOK_MAP` rewrites the host path in the event to where the
  container sees it; without it the hook fails with "reading transcript".

Set `ZEROMEM_SESSION_ID` on the server (or pass `exclude_session`) so recall
does not hand the current conversation back to itself.

### Scopes

One store can hold several projects or people. A turn carries one free-form
`scope` (`project:atlas`, `user:ben`); a turn without one is unscoped, which is
every turn written before schema 6. Recall with a `scope` returns only that
scope, matched exactly and never as a prefix; recall without one searches
everything, as it always did.

```bash
zm ingest --scope project:atlas < atlas.jsonl      # for the lines that name none
zm query "who owns billing" --scope project:atlas
zm sessions --scope project:atlas
zm hook --scope-from-cwd                           # scope = the basename of the hook's cwd
```

`ZEROMEM_SCOPE` is the default for all of the above and for `zeromem_recall`,
`zeromem_remember` and `zeromem_ingest` on the server; pass `scope: ""` on a
call to step outside it. `zeromem_stats` lists the scopes in the store.
`zeromem_read_session` never applies the default, so a turn recall returned can
always be expanded, and checks the scope only when asked to.

The scope is not part of a turn's derived uuid: re-ingesting a transcript under
a different scope is a duplicate, and the first write decides. A filter
(`scope`, `session`, `since`, `until`, `exclude_session`) is applied inside each
retrieval view, before its candidate cut, so a small scope in a large store is
not crowded out by the turns around it. A curator note takes the scope of its
sources and is refused when they span two.

### The standing brief

Everything above costs a recall. A **brief** is the part that does not: one short
paragraph per scope, written by the [curator](#curation), that a host puts in
context when a session starts. It says who and what the scope is about and what
currently holds, and nothing the stored turns do not say.

```bash
zm brief --scope project:atlas     # prints the text; prints nothing, exit 0, when there is none
zm brief --scope-from-cwd          # as a hook: scope = the basename of the hook's cwd
```

Wire it to `SessionStart` next to the `Stop` hook above; Claude Code adds what a
`SessionStart` command prints to the session's context:

```json
{ "hooks": { "SessionStart": [{ "hooks": [{ "type": "command",
    "command": "ZEROMEM_HOME=$HOME/mcp-zeromem/data zm --embedder none brief --scope-from-cwd" }] }] } }
```

An MCP client gets it without a hook: the server's `initialize` response carries
the brief of `ZEROMEM_SCOPE` as its `instructions`, dated, and the resource
`zeromem://brief/{scope}` serves any other scope (`zeromem://brief/_` is the
unscoped store's). The admin UI reads `GET /api/brief?scope=`. None of this is a
seventh tool.

A brief is a turn (`kind: brief`) that recall never returns, so writing one moves
no ranking. The brief in force is the newest one of its scope; the ones before it
stay in the store, and undoing the newest brings the previous back. Its length is
capped (Settings → Curator, 1,500 characters by default) because it is paid at the
start of every session.

## Tools

| Tool | What it does |
| --- | --- |
| `zeromem_recall` | `{query, top_k, exclude_session?, session?, since?, until?, detail, format?, context?, max_chars?}` — evidence turns with score, role (`primary`/`supporting`) and, with `detail: full`, the route taken and each turn's sources; `format: text` returns one block per hit instead, for pasting into a prompt |
| `zeromem_read_session` | `{session_id? \| around_turn?, before?, after?, limit?, offset?, format?}` — stored turns in the order they were said, either a whole session or a window centred on one turn; reports `total` and `truncated` |
| `zeromem_remember` | `{session_id, turns[{speaker, text, ts?, uuid?}]}` — reports `indexed` / `duplicates` |
| `zeromem_ingest` | Bulk JSON Lines, inline (`jsonl`) or from a file under the data directory (`path`) |
| `zeromem_stats` | Counts (turns, sessions, entities, edges, windows, episodes, embeddings), the embedder in use and whether it is the fallback; `include_sessions` adds the session list |
| `zeromem_forget_session` | Deletes a session's turns; requires `confirm: true` |

Set `ZEROMEM_READ_ONLY=true` to drop the three write tools from the listing
entirely.

A client holding the curator token also gets five `zeromem_curate_*` tools and
the `zeromem_curate*` prompts; see [Curation](#curation). Every client can read
the `zeromem://brief/{scope}` resource; see [The standing brief](#the-standing-brief).

### How recall answers

A hit is one turn, and one turn is rarely the whole answer — the question that
prompted it and the sentence that finished it are the turns either side.
`context: N` attaches up to N same-session turns on each side of every hit, as
`before` and `after` (and as `[before]` / `[after]` lines under `format: text`).
Neighbours are **attached to** evidence, never ranked **with** it: `evidence` is
identical with and without `context`, so a neighbour never takes a slot from a
real hit, and a turn that is already ranked is not repeated as another hit's
context. Hidden turns stay hidden, leaving a gap rather than being backfilled.

`format: text` keeps a turn's line breaks — a numbered list stays a list — and
cuts a turn only past `max_chars` (default `ZEROMEM_RECALL_TEXT_LIMIT`, 2000
characters). A real cut is marked, and the marker names the call that returns the
rest: `… [clipped: 2000 of 6120 characters — zeromem_read_session {around_turn:
418}]`. That matters because a model handed a fragment with no marker concludes
its memory is incomplete and goes looking elsewhere; `zeromem_read_session`
never clips, since it is what the marker points at. `format: json` has never
truncated.

The same operations are on the REST API the UI uses (`/api/status`,
`/api/sessions`, `/api/sessions/:id/turns`, `/api/recall`, `/api/recall/trace`,
`/api/ingest`), under the same bearer token.

## Configuration

| Variable | Default | Meaning |
| --- | --- | --- |
| `MCP_ZEROMEM_TOKEN` | — | Bearer token for `/mcp`. The HTTP server refuses to start without one unless `SECURE_LOCAL_NET=true` |
| `SECURE_LOCAL_NET` | — | `true` disables auth: for a trusted network only |
| `ZEROMEM_READ_ONLY` | `false` | Hide the write tools |
| `ZEROMEM_RECALL_TEXT_LIMIT` | `2000` | Characters per turn before `format: text` clips and marks the cut; `max_chars` overrides it per call (100–20000) |
| `MCP_ZEROMEM_CURATOR_TOKEN` | — | Bearer token for `/mcp` that adds the curator tools. Overrides a token set from the Settings page (see [Curation](#curation)) |
| `ZEROMEM_CURATOR` | `false` | Stdio server only: serve the curator tools |
| `DATA_DIR` | `/data` in the image, `./data` on the host | Where the store lives. Also exported to the engine as `ZEROMEM_HOME`; if both are set they must agree |
| `ZEROMEM_EMBEDDER` | `auto` | The embedder for a **new** store: `auto` loads bge-small-en-v1.5 over ONNX and falls back to a hashed embedder, loudly, if it cannot; `onnx` refuses to start instead; `hash` never downloads anything; `openai` uses the endpoint below; `none` gives this process no dense view. An existing store keeps the embedder it was built with (see [Choosing an embedder](#choosing-an-embedder)) |
| `ZEROMEM_ALLOW_EMBEDDER_SWITCH` | `false` | Force an existing store onto `ZEROMEM_EMBEDDER` at boot: drops every vector and re-embeds in the background |
| `ZEROMEM_EMBEDDING_URL` | — | Base URL of an OpenAI-compatible embeddings API, e.g. `http://npu-box:11434/v1`; required with `ZEROMEM_EMBEDDER=openai` |
| `ZEROMEM_EMBEDDING_MODEL` | — | Model name sent to that endpoint, e.g. `nomic-embed-text` |
| `ZEROMEM_EMBEDDING_API_KEY` | — | Bearer token for the endpoint. Overrides a key saved from the Settings page; never returned by the API |
| `ZEROMEM_EMBEDDING_QUERY_PREFIX`, `ZEROMEM_EMBEDDING_DOCUMENT_PREFIX` | `""` | Prepended to queries and to stored turns, for models that want them (`query: ` / `passage: ` for E5) |
| `ZEROMEM_EMBEDDING_TIMEOUT_MS` | `5000` | Per-request timeout for the endpoint |
| `ZEROMEM_MODELS` | `<DATA_DIR>/models` | Where the model is downloaded to (≈130 MB, once) |
| `ZEROMEM_SESSION_ID` | — | Session left out of `zeromem_recall` by default |
| `ZEROMEM_SCOPE` | — | Scope read and written when a call names none; see [Scopes](#scopes) |
| `EVAL_HISTORY` | `docs/eval/history.jsonl` | The eval runs the Eval page charts |
| `PORT` | `3000` | Listen port inside the container (published as 3200) |
| `HTTP_KEEP_ALIVE_TIMEOUT_MS` | `75000` | How long an idle client connection stays open (Node's own default is 5 s). Keep it above the idle timeout of any reverse proxy in front; `0` never closes one |

`GET /api/status` is unauthenticated liveness and is what the healthcheck polls.
It carries `embedder`, `embedder_is_fallback`, `embedder_active` and
`embedding_backlog`; the UI shows a banner when the fallback is in use, because
hashed vectors make dense recall a spelling match rather than a meaning match,
and a progress line while turns are waiting for a vector.

### Choosing an embedder

**The store owns the embedder.** The first process to open a store writes the
embedder it was started with into the store's `meta`; from then on every
process that opens it — the server, a stdio MCP session, a host-side `zm` —
follows the store, and a differing `ZEROMEM_EMBEDDER` is logged and ignored.
That is what keeps recall coherent when three processes share one file.

Three kinds of embedder are available:

| Kind | Name in `meta` | Notes |
| --- | --- | --- |
| Built-in BGE-small (ONNX) | `bge-small-en-v1.5` | 384 dims, in-process, ≈130 MB download on first use |
| Hash fallback | `hash-384` | Deterministic bag-of-words, no download; a spelling match, not a meaning match |
| OpenAI-compatible endpoint | `openai:<model>@<dim>` | Anything serving `POST /v1/embeddings` — Ollama, llama.cpp, vLLM, LM Studio, a box with an NPU, or the hosted API. The dimension is learned from the first response |

Change it from the **Settings** page: pick the kind, fill in the endpoint,
**Test connection** (reports the dimension and latency without touching the
store), then **Apply**. The switch is one transaction — spec written, vectors
dropped, generation bumped — and the server's embed worker re-embeds every
turn in the background in batches of 256. Recall keeps answering from the
lexical and entity views while the backlog drains; the progress bar on
Settings, the Overview and the Embeddings page shows how far along it is.
Headless hosts do the same with the CLI:

```bash
zm embedder show
zm embedder test --kind openai --url http://npu-box:11434/v1 --model nomic-embed-text
zm embedder set  --kind openai --url http://npu-box:11434/v1 --model nomic-embed-text --api-key-stdin < key.txt
zm embedder reembed    # drop every vector and re-make them with the current embedder
zm embedder drain      # re-embed the backlog now instead of waiting for the server
```

When a switch stalled half way, or the model behind an endpoint was replaced
under the same name, **Re-embed all turns** on the Settings page (or
`zm embedder reembed`) starts over with the embedder the store already names.
It is tested first, as a switch is, so an endpoint that still fails is
reported and no vector is dropped. The **Stored data** card below it clears
the store: **Clear vectors** drops every vector without testing anything
(the worker re-embeds them), and **Clear all memory** forgets every turn,
session, entity and summary, after you type `clear`. Both keep the embedder
settings, bump the generation so every other process reloads, and are
refused under `ZEROMEM_READ_ONLY` (`POST /api/settings/embedder/reembed`,
`POST /api/settings/clear` with `{"scope": "embeddings" | "memory"}`).

An API key entered on the Settings page is saved in the store's `meta` table
**in plain text**, so the stdio process and a host `zm` can use the same
endpoint. `ZEROMEM_EMBEDDING_API_KEY` in the environment overrides it wherever
it is set; the page says which one is in use and the API never returns either.
Leave the key field blank to keep the stored one.

If a process cannot build the store's embedder — no model files at hand, the
endpoint is down — it runs **without a dense view and says so**
(`embedder_active: false`, `embedder_warning` in the status), storing new turns
vectorless for the worker rather than silently substituting the hash embedder.
A remote endpoint that starts failing trips a breaker for 30 s so a recall
under load never waits out the timeout call after call.

## Seeing the store

The admin UI at `/` (same port as `/mcp`) has a page per structure the engine
keeps. Each chart has a table twin, so every number on screen can be read as
text.

| Page | What it shows | Read behind it |
| --- | --- | --- |
| Overview | Counts, embedder, turns per day and the running total | `/api/status`, `/api/viz/growth` |
| Sessions → inspect | A session's turns in order with entity spans highlighted; click one to follow it across the session, or jump to it in the graph | `/api/viz/sessions/:id/turns` |
| Graph | Force-directed entity graph: node size is degree, edge weight is shared turns; focus an entity and expand N hops; click a node for its neighbours and the turns recall finds for it | `/api/viz/graph` |
| Timeline | Sessions as lanes, windows as bands, episodes as inner segments; brush a range for the turns beneath it | `/api/viz/hierarchy` |
| Recall → Trace | One query through every stage: profile, the views it was routed to with their raw candidates, the fused list, what calibration dropped and why, the evidence | `/api/recall/trace` |
| Embeddings | 2-D PCA of the dense vectors, coloured by session; drop a query on the map to ring its nearest turns. Under the hash fallback the structure collapses, which is the point | `/api/viz/projection` |
| Compare | The same query with two settings side by side (top-k, session filters); pin side A, change B, and read the diff of the evidence sets | `/api/recall` |
| Eval | recall@k / MRR / nDCG per commit, one line per corpus × embedder | `/api/viz/eval` |
| Health | Recall latency percentiles and throughput per minute for the last hour, cold-open time, errors, then the raw status payload | `/api/viz/health` |
| Settings | The store's embedder: current name, kind and dimension, the re-embed progress, a form to test and switch to the ONNX model, the hash fallback or an OpenAI-compatible endpoint, re-embedding with the current one; and clearing the vectors or the whole memory | `/api/settings/embedder`, `/api/settings/clear` |
| Curation | Curator runs with the actions each took and why; undo one action or a whole run; the aliases and blocklist in force; the standing brief of each scope | `/api/curation/runs`, `/api/curation/actions`, `/api/curation/aliases`, `/api/brief` |

Every snapshot endpoint is capped (`limit`) so a 50k-turn store never lands
whole in a browser tab; the graph page says when it cut at the node cap. The
projection is computed in Rust and cached per generation. Health counters are
in-process and reset with the server.

The eval page reads `docs/eval/history.jsonl` (override with `EVAL_HISTORY`).
`scripts/record-eval.sh [label]` runs the harness and appends one line per
corpus × embedder, stamped with the commit; commit the file with the change
that moved the numbers.

## Curation

The store only grows: near-duplicates, pasted tool output and "ok, thanks" compete
in recall, one person is split across "Maya" and "Maya Okafor", and an old fact
ranks level with the one that replaced it. A **curator** is any MCP client, most
usefully an LLM agent on a schedule, that cleans this up. The engine stays
deterministic: it finds candidates cheaply and applies reversible operations. The
judgement, the only part that costs tokens, lives in the agent.

**Curation never deletes.** Turns are immutable; every action is logged with its
reason and can be undone from the Curation page. Purging is left to a human
(`zeromem_forget_session`, Settings → Stored data).

| Op | Effect on recall |
| --- | --- |
| `hide` / `unhide` | The turn is left out of recall (still shown, flagged, in session reads) |
| `supersede` | The old turn hands its score to its replacement, which is recalled in its place even when it shares few words with the question; the old turn is halved and folds away when both make the answer. A question about the past ("who owned … before", "… in March 2025") is left alone: the old turn is what it wants, and comes back with `valid_until`, the date its replacement was said |
| `alias` / `unalias` | Mentions of the alias count as the canonical entity in the graph and entity stats |
| `block` / `unblock` | The name is not an entity at all |
| `note` | A new turn (`kind: note`, speaker `zeromem-curator`) summarising source turns; when both make the list the sources fold under the note |
| `brief` | A new turn (`kind: brief`) that becomes the scope's [standing brief](#the-standing-brief); recall never returns it, and undoing it puts the previous brief back in force |
| `run_end` | Records the run's summary and advances the cursor the finders start from |

### The curator surface

| Tool | What it does |
| --- | --- |
| `zeromem_curate_runs` | Past runs, their action counts and summaries, the cursor, and the limits in force |
| `zeromem_curate_candidates` | `{kind, since_turn_id?, limit?, offset?}`: candidates of one kind (`duplicates`, `noise`, `aliases`, `supersession`, `consolidation`, `brief`), each with a score, a reason and a suggested op; `scanned_through` is the cursor to hand to `run_end` |
| `zeromem_curate_read` | `{turn_ids? \| session_id? \| entity?}`: turns with their flags, supersession links and covering notes |
| `zeromem_curate_apply` | `{run_id, actions[{op, …, reason}], dry_run?}`: applies a batch; each action is accepted or rejected on its own |
| `zeromem_curate_undo` | `{action_id? \| run_id?}` |

### The curator prompts

The procedures come from the server too, one MCP prompt per job, each served verbatim
from a doc so the two cannot drift:

| Prompt | Arguments | The run it describes |
| --- | --- | --- |
| `zeromem_curate` ([playbook](docs/curator-playbook.md)) | `focus?` — kinds to limit the run to | The full sweep: orient, gather candidates since the cursor, judge (when unsure, skip), dry run, apply in batches, finish with `run_end` |
| `zeromem_curate_session` ([doc](docs/curator-session.md)) | `session_id` | One conversation once it has ended: read it whole, hide its noise and repeats, supersede what it settled later, note it if it is long |
| `zeromem_curate_notes` ([doc](docs/curator-notes.md)) | `session_id?` | Consolidation on its own: find long old episodes, read them whole, write a paragraph that says only what the sources say, check it afterwards |
| `zeromem_curate_entities` ([doc](docs/curator-entities.md)) | `entity?` | Entity hygiene: fold a name into the entity it belongs to, strike out a name that is not one, each judged from the turns that mention both |

| `zeromem_curate_brief` ([doc](docs/curator-brief.md)) | `scope?` | The standing brief: find the scopes with none or with one that has fallen behind, read what is current, write one paragraph of what holds now, cite its sources |

The four focused prompts hand the sweep's cursor back to `run_end` unchanged, so a
session pass at midnight never moves the cursor past turns the nightly sweep has not
read. Under `ZEROMEM_READ_ONLY` every prompt ends with the same addendum: the write
tools are not there, so report the actions rather than applying them.

### Access and limits

Settings → Curator holds all of it:

- **The curator token.** Generate one (shown once) or paste your own. A request to
  `/mcp` bearing it gets the normal tools plus the curator surface; the normal token
  never sees them, so an ordinary agent keeps its six-tool budget. The token opens
  `/mcp` only, not the REST API. `MCP_ZEROMEM_CURATOR_TOKEN` overrides the stored one,
  and the page then says so and will not change it.
- **Serve the curator tools to every client.** Off by default; on, every `/mcp`
  client and the stdio server get them, token or not.
- **Limits.** Actions per call (default 100), actions per run (300), and a minimum
  turn age (24 h) so a live conversation is never curated under the user, and the
  longest standing brief (1,500 characters). A reason is required on every action.

`ZEROMEM_READ_ONLY` drops the curator's write tools like the others. For the stdio
server, `ZEROMEM_CURATOR=true` serves the curator tools.

### Scheduling a curator

The server runs no agent. Anything that speaks MCP can be the curator, from this
machine or another; it needs the `/mcp` URL and the curator token, no shared disk.
Notes it writes are embedded by this server.

A Claude Code scheduled task, or `claude -p` under cron, with an MCP config that
points at the store:

```json
{
  "mcpServers": {
    "zeromem": {
      "type": "http",
      "url": "http://memory-host:3200/mcp",
      "headers": { "Authorization": "Bearer ${ZEROMEM_CURATOR_TOKEN}" }
    }
  }
}
```

```cron
# Nightly at 03:30: run the playbook the server serves as its zeromem_curate prompt.
30 3 * * * claude -p "/mcp__zeromem__zeromem_curate" --mcp-config /etc/zeromem/curator.json --allowedTools "mcp__zeromem__*"
# Weekly: the jobs the sweep does least well. Both leave the nightly cursor where it was.
0 4 * * 0 claude -p "/mcp__zeromem__zeromem_curate_notes" --mcp-config /etc/zeromem/curator.json --allowedTools "mcp__zeromem__*"
30 4 * * 0 claude -p "/mcp__zeromem__zeromem_curate_entities" --mcp-config /etc/zeromem/curator.json --allowedTools "mcp__zeromem__*"
```

`zeromem_curate_session` is the one to hand a session id, from the same hook that
ingested the conversation — as long as the turns are older than `min_age_ms` by then,
which for the default 24 h means a pass the next day, not at the end of the call.

An Agent SDK script does the same from code: fetch the prompt, hand it to the agent
as its task, and let it call the tools.

```ts
import { query } from '@anthropic-ai/claude-agent-sdk';
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StreamableHTTPClientTransport } from '@modelcontextprotocol/sdk/client/streamableHttp.js';

const url = 'http://memory-host:3200/mcp';
const headers = { Authorization: `Bearer ${process.env.ZEROMEM_CURATOR_TOKEN}` };

const client = new Client({ name: 'curator', version: '1.0.0' });
await client.connect(new StreamableHTTPClientTransport(new URL(url), { requestInit: { headers } }));
const { messages } = await client.getPrompt({ name: 'zeromem_curate' });
await client.close();
const playbook = messages.map((m) => (m.content.type === 'text' ? m.content.text : '')).join('\n');

for await (const message of query({
  prompt: playbook,
  options: {
    mcpServers: { zeromem: { type: 'http', url, headers } },
    allowedTools: ['mcp__zeromem__*'],
  },
})) {
  if (message.type === 'result') console.log(message.subtype);
}
```

### A local model as the curator

`mcp-zeromem-curate` is that script for a model you host: an MCP client that hands a
curator prompt to any OpenAI-compatible `chat/completions` endpoint with tool calls
(Lemonade, llama.cpp, Ollama, vLLM) and relays the model's calls until it stops. The
server still runs no agent, and every rule the model follows is the prompt the server
serves.

```bash
ZEROMEM_CURATOR_LLM_URL=http://framework.lan:13305/api/v1   # requests go to {URL}/chat/completions
ZEROMEM_CURATOR_LLM_MODEL=qwen3.6-moe-35b-a3b-FLM

npm run curate                                  # the full sweep
npm run curate -- sweep --focus supersession    # one kind of lead
npm run curate -- session --session <id>
npm run curate -- notes
npm run curate -- entities --entity "Kenji"
npm run curate -- brief --scope project:atlas   # the standing brief; every scope that needs one without --scope
npm run curate -- sweep --dry-run               # store served read-only: it reports, changes nothing
```

With `ZEROMEM_CURATOR_MCP_URL` (and `MCP_ZEROMEM_CURATOR_TOKEN`) it curates a running
server over `/mcp`; without it, it opens the store under `DATA_DIR` in its own process,
and a note it writes is embedded by the server's worker or `zm embedder drain`. Progress
goes to stderr and a JSON report to stdout (`finished`, `closed`, steps, tool calls and
errors, tokens). The exit code is 1 when the model ran out of `ZEROMEM_CURATOR_MAX_STEPS`
(40) before finishing, so cron notices. A tool call the model gets wrong (a missing
field, a string where a boolean belongs) comes back to it as the tool's error and it
can correct itself; a long result is cut with a marker that says so.

A small model is a worse curator than a large one, and nothing here checks its
judgement. Start with `--dry-run`, then review the first real runs on the Curation page.

Review a run on the Curation page. Undoing a run restores recall to what it was.

## How recall works

Ingest writes, in one transaction, the turn, its entity spans (names, dates,
quantities, paths, code symbols and env vars found by shape, no model), its FTS5 tokens and its embedding —
or no embedding, when the writer has no dense view; every read then embeds
one batch of the backlog first, and the server's worker drains the rest.
The entity co-occurrence graph and the temporal hierarchy (sessions → windows
→ episodes, split on silences and on entity drift between windows) are
persisted next to them; appends update them incrementally, `forget` rebuilds
the aggregates from the per-turn artifacts and bumps a `generation` stamp so
every other process holding the store reloads. Opening a store is a read,
never a recomputation. The hierarchy feeds the Timeline page, `zeromem_stats`
and the curator's consolidation finder; it is not yet a recall view.

Recall returns turns, never facts distilled from them. The entity index is a
posting list — which turns mention a key, and which keys occur together — not
a table of assertions, so a hit is always something that was actually said.
That is the paper's position: a generated record costs tokens to write, and a
summary that drops a detail, merges two people or flattens an update loses the
evidence a later question needs. A superseded value stays in the store and
ranks below its replacement instead of being overwritten. The price is paid at
read time, where a turn is longer than a fact would be; `top_k`, `context` and
`max_chars` bound it, and a curator `note` is the way to condense a topic
without giving up the sources.

A query is profiled (tokens, entities, temporal cues, question form), routed
to the views that can answer it, and each view's ranked candidates are fused
by reciprocal rank. The fused list is calibrated: candidates under 0.35 are
dropped, those at 0.7 or above are `primary`, the rest `supporting`. With
`detail: full` the result carries the route and per-turn sources; the
`/api/recall/trace` endpoint (and `zm query --trace`) returns every stage.

A question about the past is read as one, from its words alone. `before`,
`previously`, `used to`, `formerly`, `originally` and the like set
`profile.history`: the value curation knows was replaced ranks first, the most
recently replaced ahead of older ones. A named period sets `profile.window` —
`2025-03-14`, `2025-03`, `March 2025`, `14 March 2025`, `in 2024`,
`yesterday`, `last week|month|year`, `3 days ago` — in UTC calendar units,
relative phrases counted from the newest turn in the store; what was in force
then ranks first: said before the period ended and not replaced before it
began, the latest such statement ahead of earlier ones. A bare month (`in
March`) is not read, since it names an event's date as often as the time
something was said. Both are preferences in the fusion, not filters, so a
store nobody curated still answers from timestamps alone and `since`/`until`
remain the way to cut. Every hit that was replaced carries `superseded_by` and
`valid_until`, the timestamp of the turn that replaced it; its own `ts` is when
it started to hold. `format: text` writes that into the hit's header.

Recall can also say that memory does not hold the answer. The names in a
question (its entities, less dates, quantities and calendar words) are what it
is about, so a turn that mentions none of them is unlikely to answer it: each
candidate's fused score is scaled by `1 − 0.3 × (the share of names its text
does not mention)`, which moves the turn about the right project above one
that only shares the question's other words. When the best hit still misses a
name, or scores under 0.5, the result carries `abstained` (`missing` lists the
names nobody mentioned, `best` the top score), `evidence` is cut to the closest
two turns, all `supporting`, and `format: text` leads with an `[abstained]`
line. A question that names nothing is never declined, and neither is an empty
result. The check has three limits worth knowing. A name written in lower case
is not seen as a name. A turn that answers without repeating the name ("she
moved it to Friday") loses up to 30% of its score, though the turn a curator
put in its place inherits its standing. And a name check cannot tell a store
that knows the project but not the asked fact from one that knows both, so
those questions are still answered.

## Numbers

Hash embedder, one core, `npm run bench` (`cargo run --release --example
bench`); CI keeps the JSON as an artifact.

| turns | ingest | cold open | RSS after open | recall p50 | recall p95 | store |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 1k | 19k turns/s | 4 ms | 19 MB | 2.2 ms | 4.1 ms | 2.4 MB |
| 10k | 14k turns/s | 58 ms | 53 MB | 18 ms | 32 ms | 24 MB |
| 50k | 9.2k turns/s | 281 ms | 126 MB | 83 ms | 121 ms | 121 MB |

Recall is linear in store size (brute-force cosine over every vector plus a
full lexical scan); an ANN index is the follow-up when stores pass 100k turns.

Eval (`cargo test -p zeromem-core --test eval`, recall@5 / MRR / nDCG@5 over
the labeled fixtures):

| corpus | hash-384 | bge-small-en-v1.5 |
| --- | --- | --- |
| small (10 queries) | 1.00 / 0.88 / 0.92 | 1.00 / 0.93 / 0.94 |
| large (266 queries) | 0.90 / 0.97 / 0.83 | 0.98 / 1.00 / 0.89 |
| transcript (446 queries) | 0.94 / 0.98 / 0.89 | 0.98 / 1.00 / 0.92 |

The floors in `tests/eval.rs` sit a little under these; raising them is how
retrieval improvements are locked in. With `ZEROMEM_EMBEDDING_URL` and
`ZEROMEM_EMBEDDING_MODEL` set the eval also scores that endpoint, recorded but
never gated, so `scripts/record-eval.sh` gives your model its own line on the
Eval page.

Those queries all ask for the current value of a fact. Each corpus also has
`probes.jsonl`, the questions they cannot ask, scored by the same test with
the hash embedder and written to `target/eval/probes/`:

| probe | what it asks | large, hash-384 | after an oracle curator |
| --- | --- | --- | --- |
| `history` | the value before the last change | 0.93 / 0.96 / 0.70 | 0.94 / 0.99 / 0.82 |
| `as_of` | the value in force in a named month | 0.81 / 0.91 / 0.71 | 0.81 / 0.91 / 0.71 |
| `abstain` | a fact the corpus never states | answered 19%, withheld 2%, AUC 0.97 | withheld 2% |

All three have floors. For `abstain`, *answered* is the share of the 58
unanswerable questions recall did not decline, *withheld* the share of the 266
answerable ones it declined by mistake, and the AUC says how far the top score
alone tells the two kinds apart (0.5 is chance). The questions still answered
name a project the corpus does talk about and ask for a fact it never states
about it. The same run reports tokens per answer, the evidence text of a top-5
recall at four characters a token: 71 on the large corpus.

To score a benchmark from outside the repo, convert it and name the directory:

```bash
cargo run -q -p zeromem-harness -- import longmemeval longmemeval_s.json --out target/longmemeval
ZEROMEM_EVAL_CORPUS=target/longmemeval ZEROMEM_SKIP_ONNX=1 \
  cargo test -p zeromem-core --test eval an_external_corpus -- --nocapture
```

The importer merges every question's sessions into one store, grades the turns
the file marks as holding the answer 2 (or the whole answer session 1 when
none is marked) and writes unanswerable questions as `abstain` probes. It
prints what it kept and what it could not place. The result lands in
`target/eval/external/`; it is never committed and no floor reads it. The
importer is tested against a hand-written sample in the published shape, not
against the dataset itself.

### Answering from memory, end to end

The numbers above say the right turn was returned. `npm run eval:answers` asks
whether a model can use it: it loads a harness corpus into a store of its own,
gives a model the two memory reads (`zeromem_recall`, `zeromem_read_session`)
and the labeled questions, and has a judge model compare each answer with the
turns the labels grade 2. For a question the store cannot answer, the right
reply is "I don't know."

```bash
ZEROMEM_EVAL_LLM_URL=http://framework.lan:13306/v1    # the model under test; it also grades,
ZEROMEM_EVAL_LLM_MODEL=Qwen3.6-35B-A3B                # unless ZEROMEM_EVAL_JUDGE_LLM_* names another

npm run eval:answers -- crates/zeromem-harness/fixtures/small
npm run eval:answers -- crates/zeromem-harness/fixtures/large --limit 60 --out answers.json
npm run eval:answers -- crates/zeromem-harness/fixtures/small --curate   # the curator model sweeps first
```

The report gives, per kind of question, how many were right, wrong, declined or
cut off at the reply cap (`ZEROMEM_EVAL_LLM_MAX_TOKENS`, 4096) before the model
answered, with the tool calls and prompt tokens an answer took. Both ends are
models and the grader here is the model under test, so the number moves with the
model and is recorded, never gated. `--curate` is the only measure there is of a
model curator, as opposed to the oracle one in `tests/eval.rs`.

One run, Qwen3.6-35B-A3B answering and grading, hash-384, 60 questions spread
over the large corpus and its probes:

| question | asked | right | wrong value | declined | cut off | prompt tokens / answer |
| --- | --- | --- | --- | --- | --- | --- |
| current value | 27 | 25 | 2 | 0 | 0 | 4,538 |
| history ("before") | 11 | 6 | 2 | 0 | 3 | 4,897 |
| as of a date | 17 | 8 | 3 | 3 | 3 | 8,877 |
| not in the store | 5 | 4 | 0 | 4 | 1 | 23,390 |

Both current-value misses named an older value of a fact that changed four or
more times. A history question counts only the value just before the current
one, so naming an earlier one is wrong. A question the store cannot answer
costs five times the tokens of one it can, because the model keeps searching
before it gives up.

### Comparing against upstream

The upstream binary is a black box here (see the clean-room policy). To
compare rankings, run both over the same fixture turns and score the lists:

```bash
scripts/rank.sh large 5 > ours.jsonl          # our answers, one {"id","ranked":[uuid…]} per line
# …run the other ranker over crates/zeromem-harness/fixtures/large/turns.jsonl
#    and emit the same shape as theirs.jsonl
cargo run -q -p zeromem-harness -- compare large ours.jsonl theirs.jsonl
```

That prints both sides' metrics against the labels, plus top-k overlap and
Spearman rank agreement. The adapter that drives upstream's binary is not in
this repo yet; CI uploads `ours.jsonl` for every commit so it can be run
against any version. The comparison is tracked, never gated: the labels are
the gate.

## Layout

```
crates/zeromem-core/   the engine (Rust lib) + the `zm` CLI; tests/ holds the oracle + properties
crates/zeromem-harness/ corpus generator, labeled queries, eval metrics; fixtures/ is generated
crates/zeromem-node/   napi-rs addon → @mcp-zeromem/native
shared/src/            zod DTOs shared by server and app
server/src/            Express 5: /api routes (viz/ for the charts), /mcp gateway, stdio bin, metrics.ts
app/src/               React admin UI; routes/ is one file per page, components/viz/ the chart kit
docs/curator-*.md      the curator procedures, served verbatim as the MCP prompts
docs/eval/             history.jsonl, the recorded eval runs the Eval page reads
scripts/               smoke-stdio.sh, rank.sh, record-eval.sh
```

## Development

```bash
npm run lint && npm run typecheck && npm test && npm run test:rust
npm run build            # addon → shared → server → app
./scripts/smoke-stdio.sh # JSON-RPC over the stdio bin
npm run bench            # cold open / RSS / recall latency at 1k, 10k, 50k turns
scripts/record-eval.sh   # append this commit's eval metrics to docs/eval/history.jsonl
npm run routes:gen -w app # regenerate app/src/routeTree.gen.ts after adding a route file
```

Charts follow the `dataviz` house rules: a fixed eight-slot categorical
palette (`--viz-series-1..8` in `app/src/index.css`, validated for both
themes) that is never cycled, one axis per chart, thin marks, a legend for
two or more series, a table view for every chart, and the previous render
held at reduced opacity while a refetch is in flight.

`npm run test:rust` downloads the embedding model into `target/models` on
first run (set `ZEROMEM_SKIP_ONNX=1` to skip the ONNX halves of the
reference-vector and eval tests). `UPDATE_GOLDEN=1` rewrites the retrieval
snapshots under `crates/zeromem-core/tests/golden/` after an intended ranking
change; review the diff.

The repo is a clean-room rewrite: no code, scripts, fixtures or schema from the
upstream project are used, and none may be added. See the policy in
[`AGENTS.md`](AGENTS.md).