# Distributed Memory System

**Cross-project memory for [Claude Code](https://claude.com/claude-code) — a learning written
while working in one project becomes recall-able in _every_ project, without leaking
project-specific facts or secrets.**

Claude Code's native memory is keyed per repository: what you learn in project A is invisible
in project B. This system layers a shared, git-versioned markdown store plus local semantic
recall on top — keeping the good per-project notes you already write, while making the
*reusable* ones available everywhere, and enforcing at the write path that secrets and PII
never escape their scope.

It is small (617 lines of Python across 8 files), fully local (no API calls at runtime), and
the entire index is a disposable cache you can delete and rebuild from the markdown at any time.

---

## It works — and here's the receipt

This isn't a demo. The store below is live. It was **seeded once** by migrating a real
engineer's accumulated Claude Code memory (24 project stores + a cross-project engineering
knowledge base + global instructions → 203 raw candidates → **150 gated, de-duplicated
memories**), and has since been **growing on its own** as that engineer works:

| | Count |
|---|---:|
| **Total memories** | **160** |
| — seeded by migration | 150 |
| — **captured organically since activation** (`memory_write` during real sessions) | **10** |
| Global (cross-project) | 74 |
| Project-scoped | 86 across **32 project scopes** |
| By type | project 70 · reference 61 · procedural 21 · feedback 5 · user 3 |
| By domain | ai 64 · infra 29 · db 23 · design 18 · general 16 · bioinfo 10 |
| By sensitivity | internal 83 · public 75 · **secret 2** (both project-scoped, redacted at rest) |

**The value loop, observed in the wild.** In the hours after activation, an engineer rebuilt a
Next.js app ("Aurelia") following the built-in sprint workflow. The system captured, unprompted:
a deep-research brief, a spec-first test surface, testing gotchas — and five *generalizable*
gotchas it correctly promoted to **global** scope, e.g.:

> **`drizzle_neon_http_has_no_interactive_transactions_use_neon_serverless`** — "Drizzle
> `neon-http` throws on `db.transaction()`; use `neon-serverless` (WebSocket) for atomic writes
> on Vercel. Local dev = `node-postgres`; prod = `neon-serverless`; only the driver import swaps."

That memory was learned in **one** project. Its body ends with `[[nextjs_local_node_postgres_not_neon_http]]`
— an automatic link to a memory that was migrated from a **different** project (TubeIntel).
Two projects, two points in time, **one connected knowledge graph.** That is the whole thesis in
a single artifact.

There's even a memory the system wrote *about itself* — `use_mcp_memory_db_not_memory_md` — a
learning that the MCP store, not scattered `MEMORY.md` files, is now the source of truth.

---

## Value propositions

- **Cross-project recall.** A gotcha, convention, or decision written anywhere is retrievable
  everywhere — by *meaning*, not just keywords.
- **Security enforced on the write path, not the reader's discretion.** Every write is scanned
  for secrets, PII, and machine-local paths. `global` scope is *provably* free of secrets and
  PII (gate-enforced); project scope may hold sensitive facts, but secrets are **redacted at
  rest**. A pre-commit hook is the belt to the gate's suspenders.
- **Local and offline.** Embeddings run on-device ([fastembed](https://github.com/qdrant/fastembed),
  ONNX/CPU). No data leaves the machine; no per-query API cost.
- **Markdown + git is the source of truth.** Every memory is a plain, diffable, reviewable file.
  The search index is a *derived* SQLite cache — delete it and rebuild from the markdown at any time.
- **Scoped, not global-by-default.** `global` for reusable-everywhere knowledge; `project:<name>`
  for the rest. Promotion (project → global) is **propose → approve**, with provenance retained.
- **Auditable.** 617 lines of Python, no framework magic, every decision inspectable.

---

## Architecture

Two planes: a **source-of-truth plane** (markdown in git) and a **derived-index plane** (SQLite).
The index is always reconstructible from the source; the source never depends on the index.

```mermaid
flowchart TB
    subgraph SOT["Source of truth — versioned in git"]
        MD["store/**/*.md<br/>one memory = one file<br/>frontmatter + body"]
    end
    subgraph DERIVED["Derived index — gitignored, rebuildable"]
        FTS["memories_fts<br/>SQLite FTS5 · BM25"]
        VEC["embedding BLOBs<br/>384-dim float32 · normalized"]
    end
    subgraph ENGINE["server/ — 617 LOC"]
        IDX["indexer.py<br/>content-hash cached"]
        SRCH["search.py<br/>hybrid + MMR"]
        GATE["gate.py<br/>sensitivity scanner"]
        MCP["mcp_server.py<br/>5 tools over stdio"]
    end
    CC["Claude Code<br/>(any project)"]

    MD -->|"parse + embed<br/>(only changed files)"| IDX --> FTS & VEC
    CC <-->|"memory_search / memory_write / …"| MCP
    MCP -->|recall| SRCH
    SRCH -->|reads| FTS & VEC
    MCP -->|"write path"| GATE -->|"redacted, scoped"| MD
    MCP -.->|"reindex once after write"| IDX
```

### The write path — where security lives

Secrets are contained at the source, not left to the reader:

```mermaid
flowchart LR
    W["memory_write<br/>(title, body, scope…)"] --> SCAN["gate.scan()<br/>7 secret · 2 PII · 2 path rules"]
    SCAN --> CHK{"scope == global<br/>AND secret/PII?"}
    CHK -->|yes| BLOCK["❌ blocked<br/>'use a project scope'"]
    CHK -->|no| RED["redact_secrets()<br/>mask value, keep structure"]
    RED --> ESC["escalate sensitivity<br/>to match findings"]
    ESC --> FILE["write store/…/slug.md<br/>UTF-8, with frontmatter"]
    FILE --> RI["reindex once"]
```

- **`global` scope is gate-enforced clean.** A write to `global` containing a secret or PII is
  *rejected*, not silently downgraded. (Verified across all 74 global memories in this store: zero
  secret/PII findings.)
- **Redact at rest, keep the shape.** A DSN like `postgresql://user:pass@host/db` is stored as
  `postgresql://user:«REDACTED:db-password»@host/db` — the locator survives, the secret doesn't.
- **Sensitivity auto-escalates.** If you label a memory `public` but the scanner finds PII, it's
  promoted to `internal` before it's written. You can't under-classify by accident.

### The read path — hybrid recall

Semantic similarity finds *what you meant*; keyword search anchors *exact terms* (library names,
error strings, flags). Blended, min-max normalized, then diversified with MMR so the top-k aren't
near-duplicates:

```
score = 0.7 · cosine(query, memory)      # vector — semantic
      + 0.3 · bm25(query, memory)         # FTS5  — lexical
```

Real query against the live store — *"drizzle neon transaction on vercel"*:

| memory | score | vec | bm25 |
|---|---:|---:|---:|
| `drizzle_neon_http_has_no_interactive_transactions…` | **1.00** | 1.00 | 1.00 |
| `pglite_hermetic_drizzle_postgres_integration_tests` | 0.67 | 0.76 | 0.46 |
| `tubeintel_stack_and_architecture` | 0.55 | 0.60 | 0.43 |

The exact learning surfaces first, with a related testing memory and the originating project's
architecture right behind — the shape you want for "remind me what I know about X."

---

## Design decisions & nuance

The interesting engineering is in the *why*. Each choice below traded something for something.

**1. Markdown + git is truth; SQLite is a cache.**
Memories are durable, human-reviewable, and diff cleanly in PRs. The index (FTS5 + embedding BLOBs)
is gitignored and disposable: `rm index.db && python server/indexer.py` fully rebuilds it. This
means no schema migrations to fear, no lock-in, and a store you can hand-edit or grep. The cost is
a rebuild step — paid for by content-hash caching (below).

**2. Content-hash caching — only re-embed what changed.**
Each file's SHA-256 is stored alongside its vector. On reindex, an unchanged file reuses its cached
embedding; only new/edited files hit the model. The bulk migration re-embedded 150 files once;
every subsequent `memory_write` re-embeds exactly one.

**3. Sensitivity gate on the _write_ path, not the read path.**
Filtering secrets at read time trusts every reader forever. Gating at write time means the
dangerous data never lands in a shareable scope in the first place — and the guarantee is
structural, not behavioral. The gate is deliberately layered: the MCP tool calls it, *and* a
portable git pre-commit hook re-runs it on anything added outside the tool.

**4. The gate is heuristic — and was hardened by real data.**
The shipped patterns caught `password:`-style secrets and `/home/` paths. Mining a real corpus
exposed two blind spots it would have leaked to `global`: **DSN-embedded passwords**
(`postgresql://user:pass@…`) and **Windows absolute paths** (`C:\…`). Both are now rules
(7 secret · 2 PII · 2 path). This is the honest posture: a gate is a strong default, not a DLP
guarantee — so it's designed to be extended, and it was.

**5. Scopes + propose→approve promotion.**
`global` knowledge is small, clean, and always-on; `project:<name>` knowledge is abundant and may
be sensitive. Promotion is never automatic: `memory_promote` returns a gate verdict and a proposed
move, but a human approves it. Provenance (`source_project`) is retained so a global memory always
remembers where it was learned.

**6. Hybrid retrieval + MMR, with fixed, legible weights.**
0.7/0.3 vector/BM25 and MMR λ=0.7 are constants, not a tuned model — chosen because they're
explainable and good enough, and because a memory system's failure mode should be "returned
something slightly off," never "silently mis-ranked by an opaque scorer." Both signals are
min-max normalized per query so neither dominates by scale.

**7. Local 384-dim embeddings (`bge-small-en-v1.5`).**
384 dimensions (1536 bytes/vector) is the sweet spot for a personal store of hundreds–thousands of
memories: strong retrieval, tiny footprint, fast on CPU, no API dependency. Vectors are L2-normalized
so cosine similarity is a single dot product.

**8. stdio protocol discipline.**
An MCP stdio server must never write to stdout — it corrupts the JSON-RPC stream. The indexer prints
progress, so the server redirects its stdout to stderr around every reindex. Small detail, total
protocol failure if missed.

**9. UTF-8, always.**
The store legitimately contains em-dashes, arrows, and the `«REDACTED»` guillemets. Every file read
and write pins `encoding="utf-8"` — because relying on the platform default (cp1252 on Windows)
crashes `memory_write` the moment a memory contains one of those characters, and corrupts git blobs
even when it doesn't.

---

## The always-on layer

The store holds hundreds of memories, but only a lean **index** is ever loaded into a session —
full bodies are fetched on demand via `memory_search`. `regen_rules.py` generates
`~/.claude/distributed-memory.md`: the usage protocol plus a one-line entry per global memory,
imported by `~/.claude/CLAUDE.md`. This keeps every session cheap while making the whole store
reachable in one tool call.

```
## Global memories (74) — one-line index; call `memory_search` for full detail
- drizzle_neon_http_has_no_interactive_transactions… [db] — Drizzle neon-http has no interactive
  transactions; use neon-serverless (WebSocket) for atomic writes on Vercel
- fastapi_spa_mount_order [infra] — Mount StaticFiles(html=True) LAST, after every API/WS route…
- …
```

---

## Schema

One markdown file = one memory. Frontmatter + body. Compatible with Claude Code auto-memory
(`name` / `description` / `metadata.type`) and extended for cross-project recall.

```markdown
---
name: <slug = filename stem>
description: "<one sharp sentence — this is what recall ranks on>"
metadata:
  type: user | feedback | project | reference | procedural
  scope: global | project:<name>
  domain: db | bioinfo | ai | design | infra | general
  sensitivity: public | internal | secret
  source_project: <origin repo>          # provenance, retained through promotion
  origin_session: migration | mcp | …
  created: <YYYY-MM-DD>
---
<body — "Why:" / "How to apply:" encouraged; [[links]] to related memories>
```

- `description` is **load-bearing**: recall ranks on it and the always-on index shows it verbatim.
- `scope: global` MUST be secret/PII-free (gate-enforced) and should avoid machine-absolute paths.
- `sensitivity: secret` never leaves project scope.
- `[[name]]` links reference another memory's slug — this is how the knowledge graph forms.

Average body length in the live store: **~506 characters** — sharp and single-fact, not essays.

---

## MCP tools

Registered once at user scope, available in every project:

| Tool | Purpose |
|---|---|
| `memory_search(query, scope?, domain?, project?, k?, max_sensitivity?)` | Hybrid recall by meaning + keywords. |
| `memory_write(title, body, type, domain, sensitivity, scope\|project, links?)` | Gated write; redacts secrets, escalates sensitivity, reindexes. |
| `memory_promote(name)` | **Proposal only** — returns the gate verdict for a project→global promotion; a human approves. |
| `memory_reindex()` | Rebuild the index from the markdown store. |
| `memory_status()` | Counts by scope/domain + the embedding model. |

> MCP stdio servers must never write to stdout — it corrupts the protocol. Keep all logging on stderr.

---

## Component map

| File | LOC | Role |
|---|---:|---|
| `server/embed.py` | 21 | Local embeddings (fastembed, ONNX/CPU) |
| `server/mem.py` | 42 | Frontmatter + body parsing, content hashing |
| `server/precommit.py` | 55 | Pre-commit sensitivity gate (belt-and-suspenders) |
| `server/regen_rules.py` | 59 | Generate the always-on protocol + global index |
| `server/gate.py` | 78 | Secret / PII / path scanner + redactor |
| `server/indexer.py` | 80 | Build/refresh `index.db`, content-hash cached |
| `server/search.py` | 119 | Hybrid BM25 + vector recall, MMR-diversified |
| `server/mcp_server.py` | 163 | MCP server: the 5 tools, over stdio |
| **Total** | **617** | |

---

## How the initial corpus was built (a case study in itself)

The 150 seed memories weren't hand-written — they were **mined** from an engineer's real,
scattered memory artifacts, which is a nice demonstration of the multi-agent workflow the system
now recommends:

1. **Fan-out mining** — 6 parallel agents read 40+ source artifacts (24 project `MEMORY.md` stores,
   a large cross-project knowledge base, global instructions, in-repo CLAUDE.md files) and extracted
   **203 candidate memories** in a structured schema.
2. **Normalization** — duplicates merged (e.g. `python -m pip` appeared 4×), a fragmented user
   profile consolidated, and a name collision resolved (two distinct projects both called "NEXUS"
   → separate scopes).
3. **Security triage** — a shared Postgres credential found across ~10 projects was collapsed into a
   single `secret` memory (password omitted, since it was being rotated); ~20 machine paths kept in
   project scope but stripped from every global.
4. **Gated bulk write** — every candidate passed through the same `gate.py` used by `memory_write`,
   then a single reindex — with a post-write assertion that **no global memory carries a secret or PII**.

The full step-by-step process — the six mining agents (with telemetry), the consolidation
decisions, the security triage, and verification — is documented in
[`docs/MIGRATION.md`](docs/MIGRATION.md).

---

## Quick start

```bash
git clone <this-repo> && cd claude-memory-system
python -m venv .venv

# Windows:            .venv/Scripts/python -m pip install -r requirements.txt
# macOS/Linux:        .venv/bin/pip install -r requirements.txt

# Build the index from the markdown store (first run downloads the embedding model, then offline)
.venv/Scripts/python server/indexer.py        # or .venv/bin/python

# Try a recall
.venv/Scripts/python server/search.py "postgres transaction on vercel"

# Install the sensitivity pre-commit gate (portable; detects .venv on Windows or POSIX)
cp hooks/pre-commit .git/hooks/pre-commit      # or: ln -sf ../../hooks/pre-commit .git/hooks/pre-commit

# Wire the MCP server into Claude Code (user scope = available in every project)
claude mcp add memory -s user -e PYTHONUTF8=1 -- \
  /abs/path/.venv/Scripts/python.exe /abs/path/server/mcp_server.py

# Generate the always-on protocol + global index, and import it
.venv/Scripts/python server/regen_rules.py
```

---

## Limitations & honest trade-offs

Good engineering names its edges:

- **The gate is a strong heuristic, not a DLP system.** It catches common secret/PII/path shapes and
  is easy to extend, but a novel secret format can slip past. `global` scope is the hard boundary;
  treat project scope as "may contain sensitive facts."
- **Single-user by design.** No auth, no concurrency control beyond git. It's a personal/pair store.
- **Retrieval weights are fixed defaults**, not learned. Legible and good enough; not SOTA ranking.
- **The always-on global index grows with the global count.** At 74 lines it's cheap; a store with
  thousands of *global* memories would want tiering. Project memories don't have this cost — they're
  fetched only on demand.
- **384-dim embeddings** favor footprint and speed over the last few points of retrieval accuracy a
  larger model might buy.

---

## Model

markdown + git = source of truth · SQLite = derived, disposable index · local offline embeddings ·
hybrid vector + BM25 recall, MMR-diversified · sensitivity gate on the **write** path · scoped, with
human-approved promotion · 617 lines, fully auditable.

## License

MIT — see [LICENSE](LICENSE).