claude-memory-graph

Persistent, queryable memory over my own Claude Code history: a knowledge graph plus a vector index over past conversations, exposed as an MCP server so Claude can recall my own history mid-session.

Everything runs locally except two API calls (entity extraction and embeddings). No framework, no ORM, no vector database — Python 3.14 standard library, one SQLite file, 27 MB.

Headline result, in two parts.

The hybrid retrieval this project set out to build — fixed-weight RRF over vector and graph — loses to both of its own inputs. Vector and graph turn out to be strongly complementary with neither dominating, and blending them at a fixed ratio is worse than picking one.

Replacing fusion with routing fixes it. Letting vector's own confidence decide how much graph to mix in gets overall MRR 0.438 vs 0.353 for the best fixed mode (+24%), and matches graph's cross-session recall (hit@10 0.85) while keeping almost all of vector's single-chunk recall (0.84 vs 0.88).

The negative result came first and is what produced the working one, so both stay at the top rather than only the flattering half. Numbers in Evaluation.

What it does

~/.claude/projects/**/*.jsonl          25 sessions, 2026-05-26 → 2026-09-11
Claude Desktop session store           session titles
<project>/CLAUDE.md                    curated project context
          │
          ├─ ingest.py        normalize + chunk        →  411 chunks, 1.1M chars
          │                                               3,075 structural refs
          ├─ extract.py       LLM triple extraction    →  2,126 triples
          ├─ graph.py         + structural edges       →  2,579 entities
          │                     prune, renormalize        2,895 triples total
          ├─ embed.py         chunk vectors            →  411 × 1536-d
          ├─ embed_entities.py entity-name vectors     →  2,579 × 1536-d
          │
          ├─ retrieve.py      auto | vector | graph | hybrid
          └─ server.py        MCP: search_memory, get_related_entities, add_memory

Related MCP server: Loc Knowledge Graph Memory Server

Quick start

cp .env.example .env          # add SEGMIND_API_KEY
python ingest.py              # transcripts → memory.db
python extract.py             # triples (≈32 min, ≈0.14 Segmind credits)
python graph.py --build       # structural edges + prune + renormalize
python embed.py               # chunk vectors (≈7 min)
python embed_entities.py      # entity vectors (≈25 min)
python retrieve.py "what blocked the model eval?"           # auto mode
python evaluate.py --ablate                              # per-set tables
python benchmark_all.py                                  # the combined table above

Every module has a runnable self-check with no framework or fixtures:

for m in ingest extract graph embed retrieve server; do python $m.py --selfcheck; done
# ingest extract graph embed retrieve server -> 6/6 ok

Data sources

Claude Desktop is installed as an MSIX/Store package, so its AppData is virtualized — %APPDATA%\Claude does not exist and the real store lives at:

%LOCALAPPDATA%\Packages\Claude_pzs8sxrjxfjjc\LocalCache\Roaming\Claude\

What is actually there, after checking rather than assuming:

claude.ai chat history is server-side only. The desktop store contributes titles, not conversations.

Chunking

One chunk = one exchange (a human prompt plus the replies it drew), split at 6,000 chars on paragraph boundaries. The exchange is already the coherent unit — one intent, one resolution — so no topic-shift heuristic was needed.

98% of the bytes are tool-call payloads. They are dropped as text but mined for 3,075 structural refs (file, command, mcp, url, search), which become graph edges that cannot be hallucinated because no model produced them.

Graph schema

Closed vocabulary, enforced at validation — anything outside it is dropped rather than stored. Freeform extraction invents a new predicate per sentence and the graph stops being traversable.

Entity kinds — artifact, concept, technology, project, problem, decision, person Predicates — uses, implements, evaluates, causes, fixes, replaces, depends_on, compared_to, part_of, blocked_by, decided

triples.src separates llm (2,126) from structural (769) so the eval can ablate them.

Evaluation

Two eval sets, because one of them alone would have been misleading.

Neither is neutral. Both are reported.

Single-hop recall — 25 questions

mode                     hit@1   hit@3   hit@5  hit@10     MRR    p50ms  compute
vector                    0.44    0.56    0.88    0.88   0.576      939    139.8
graph                     0.24    0.36    0.40    0.56   0.323     2033   1080.9
hybrid                    0.40    0.48    0.60    0.76   0.482     3228   1225.1
graph (lexical seeds)     0.16    0.20    0.20    0.28   0.191      307    307.1
hybrid (lexical seeds)    0.24    0.28    0.60    0.76   0.349     1405    476.2

Cross-session recall — 20 questions

mode                     hit@1   hit@3   hit@5  hit@10     MRR    p50ms  compute
vector                    0.05    0.10    0.10    0.15   0.074     1452    149.8
graph                     0.20    0.45    0.55    0.75   0.344     2121   1090.5
hybrid                    0.05    0.35    0.35    0.60   0.216     3227   1177.0
graph (lexical seeds)     0.25    0.50    0.65    0.85   0.420      307    307.3
hybrid (lexical seeds)    0.05    0.40    0.40    0.65   0.234     1170    421.1

compute is latency with embedding-API time subtracted. The Segmind call ranged 0.6–15 s for identical requests and dominates wall clock, so end-to-end latency mostly measures Segmind's load, not this code.

Routing, and why it exists

Fixed fusion failing is what motivated auto. All four modes, both sets, lexical seeds throughout, 45 questions total:

mode        single MRR  multi MRR  overall MRR  single h@10  multi h@10
vector           0.576      0.074        0.353         0.88        0.15
graph            0.191      0.420        0.293         0.28        0.85
hybrid           0.349      0.234        0.298         0.76        0.65
auto             0.538      0.313        0.438         0.84        0.85

auto runs vector first and lets its top cosine set the graph weight:

top cosine >= 0.55  ->  graph weight 0.0   (skip traversal entirely)
top cosine <= 0.40  ->  graph weight 3.0
between             ->  linear

The reasoning: when the nearest chunk vector is far from the query, no chunk states the answer, so the answer is likelier reachable than stateable. Continuous rather than a hard switch, so borderline queries degrade gracefully; and at high confidence it skips traversal, making confident queries faster than fixed hybrid rather than slower.

What the numbers say

1. Fixed hybrid loses on both sets — 0.349 vs vector's 0.576 single-hop, 0.234 vs graph's 0.420 multi-hop. Unweighted RRF mixes a weak ranking in at equal strength and drags the strong one down.

2. Routing recovers it. Overall MRR 0.438 vs 0.353 for the best fixed mode. Recall is the clearer story: auto matches graph's multi-hop hit@10 exactly (0.85) and keeps nearly all of vector's single-hop hit@10 (0.84 vs 0.88). It does not beat each specialist's MRR on that specialist's home set (0.538 vs 0.576; 0.313 vs 0.420) — ranking is slightly diluted, coverage is near-best on both.

3. Graph beats vector 5× on cross-session recall (hit@10 0.85 vs 0.15) and loses badly on single-chunk recall. Complementarity, not superiority.

4. Semantic vs lexical seeding flips between sets. Semantic wins single-hop (0.323 vs 0.191), lexical wins multi-hop (0.420 vs 0.344) and is 7× faster. The multi-hop questions name the bridge entity verbatim, which suits exact matching — a construction artifact, not a general result.

5. Latency: graph traversal is ~300 ms with lexical seeds and no network. Vector needs one embedding call. Fixed hybrid pays for both and wins neither.

The routing result is overfit, and by how much

AUTO_HI, AUTO_LO and AUTO_MAX_W were fit on the same 45 questions reported above. That is textbook overfitting on a small sample from one person's history. The honest claim is "this works on my corpus", not "this generalises".

A rejected alternative is worth recording: routing on the phrase "where else / other project" would have scored better still — and would have been meaningless, because make_multihop.py instructs the generator to use that phrasing. The router would have been detecting my own prompt. Vector self-confidence was chosen because it is a property of the retrieval, not of how the questions were written.

Caveats that bound all of the above

• 411 chunks, 25 sessions, 23 active days. Small.

• One gold chunk per question understates recall when a fact appears in several chunks. It penalises every mode equally, so comparisons hold and absolute numbers read pessimistically.

• Questions are LLM-generated from real chunks, not hand-written. The generator saw only chunk text — never the graph, the embeddings, or any retriever — so it cannot favour one retriever, but it does inherit the gold chunk's phrasing.

• The single-hop set was built first, and I only recognised it favoured vector after vector won. That is post-hoc, which is why the mirror-image set exists.

• Some questions still leak literal strings ("video-04", "336 px") despite the prompt forbidding it, partially reducing those to string matching.

What didn't work

Every embedding was identical and the benchmark looked fine. Segmind's embedding endpoint takes prompt; I sent OpenAI-style input. The unknown field is ignored and a well-formed 1536-d vector comes back — for a default string. All 411 chunks embedded to the same vector, every cosine was exactly 1.0, and vector search returned chunks in arbitrary order while reporting plausible results. I caught it only because --explain printed cos=1.0 on every row. verify_api() now embeds two unrelated sentences and refuses to build the index if they score above 0.9 (real value: 0.076).

Three graph bugs, all mine, all silent:

• build_structural collapsed paths to basename, fusing VAD/README.md with chat-connect/README.md into one node spanning 4 projects — fabricating exactly the multi-hop paths the project claims to find. Generic filenames are now project-qualified.

• cd reached degree 159 across 7 projects. Every project runs cd, so one node became adjacent to everything. Navigation commands and shell keywords (for, foreach — the ref extractor was reading loop syntax as tool names) excluded.

• The extractor wrote false_alarm_rate while queries say "false alarm", so the graph held the answer and could never reach it. Normalising _/- to spaces merged 52 duplicate nodes and turned a miss into an exact hit.

Hub explosion. one project node has degree 347; two hops reached the whole project and traversal returned a blob. Traversal now refuses to expand through high-degree nodes — including seed nodes, which was the version that mattered.

A 3-question probe lied. It showed semantic seeds beating lexical 0.583 vs 0.333 and I used it to justify 2,579 entity embeddings. At full scale the result splits by eval set. Three questions is a smoke test, not evidence.

Cost of a masked failure. The first extraction run burned credits and committed nothing: pool.map yields in submission order, commits fired every 25 results, and stdout was buffered. Now as_completed, commit per result, unbuffered — a kill cannot discard paid-for work.

Design decisions

SQLite, not Neo4j. Docker is not installed and the machine has ~890 MB free RAM of 7.5 GB. At 2,579 nodes, traversal is a recursive CTE taking ~300 ms; Neo4j would buy resume signalling, not performance. graph.py keeps edges in a plain table, so a Neo4j export is ~30 lines if it is ever worth it.

One LLM call per chunk, not batched. gpt-5-nano costs ≈0.000345 credits/call, so batching would trade exact provenance for a saving too small to measure.

text-embedding-3-small, not large. 1536-d at 0.9 s vs 3072-d at 7.2 s. At 411 chunks the extra dimensions are not measurable; the 8× latency is. Note: passing a list to the embedding endpoint silently returns only the first element's vector, so Embedder raises TypeError on a list rather than letting a batching "optimization" corrupt the index.

Pure-Python cosine. 411 × 1536 pre-normalized float32 vectors, dot product over array('f'), ~50 ms. An ANN index would be slower at this size.

MCP server

"memory-graph": {
  "command": "C:\\Python314\\python.exe",
  "args": ["D:\\Projects - AI ML LLMs\\personal brain\\server.py"],
  "env": {"PYTHONIOENCODING": "utf-8"}
}

Verified over a real stdio transport (initialize → tools/list → tools/call), not just by importing the module. MCP 2.x renamed FastMCP to MCPServer.

add_memory embeds the note and refreshes the in-memory index so it is searchable in the same session; if embedding fails it rolls back, because a chunk with no vector is invisible to search and worse than not saving at all.

Known limitations

• get_related_entities spends its result budget on seed variants when semantic seeding returns many near-identical matches (Remotion, Remotion Studio, Remotion pipeline). Seeds should be capped separately from results.

• add_memory does not extract triples — the note is vector-searchable at once but reaches the graph only on the next extract.py run.

• Reindexing is manual. No file watcher, no cron.

• The corpus has real cross-project tool reuse (Remotion, React, Python, Google Fonts) but thin cross-project concept reuse — the same kind of work was rarely done twice. That bounds how much any graph method can recover.

• auto uses lexical seeds throughout, because they win on the multi-hop set. Semantic seeds win on single-hop, so seeding could itself be routed. Untested.

• 45 questions is too few to separate a 0.02 MRR difference from noise. Treat only the large gaps (0.15+) as real.

A note on the examples

Project names in this README are anonymized (project-a, video-04). Every number, bug and behaviour described is real and came from the author's own corpus; only the project labels are placeholders. The eval question sets are not published — they are generated from private conversations — but the harness that builds them (make_evalset.py, make_multihop.py) and every result file are, so the method is reproducible against your own history.

Privacy

Personal use only. Transcripts include unpublished research and patent material. memory.db, .env and api.txt are gitignored. Chunk text does leave the machine for entity extraction and embedding (Segmind); nothing else does.