semantic-memory-mcp

English | 中文

A sidecar for coding-agent context management: model-semantic-level context retrieval that keeps context relevance alive.

Its goal is not saving tokens or compressing context. opencode v2's native compaction and DCP manage context quantity (compress, prune, discard to fit the window); this component manages context relevance — everything that gets compressed away, pruned, or left behind in a previous session lands in a semantically searchable store, and is automatically recalled and injected when it becomes relevant to the current task. It spends a small, budgeted amount of tokens (the injection block) to buy back relevance that would otherwise be lost forever. Compaction makes the window fit; the sidecar makes compression safe.

• MCP server (Python FastMCP): BM25 (dual-channel FTS5: porter + trigram) + vectors (Qwen3-Embedding-0.6B GGUF, optional) + RRF fusion — retrieval happens in embedding space, so differently-phrased memories still match

• opencode v2 plugin (opencode-plugin/memory.ts): hooks into context / compaction / generate / chat.message / tool execute.after, making remembering and recalling automatic

• Separate process, separate model: never touches the main agent's window or model; if the service is down the session is unaffected (fail-open)

Architecture

┌─ opencode v2 plugin (memory.ts, zero deps, fail-open) ───┐
│  context/compaction/generate → /recall → tail injection  │
│  pre-compaction → /ingest (discarded ≠ lost)             │
│  chat.message "remember ..." → /ingest                   │
│  tool execute.after (whitelist) → /ingest                │
└──────────────────────┬───────────────────────────────────┘
                       │ HTTP 127.0.0.1:8765 (optional Bearer token)
┌──────────────────────▼───────────────────────────────────┐
│  memory-mcp service (single process, dual interface)     │
│  MCP /mcp: memory_store / recall / forget / list / stats │
│  REST sidecar: /recall /ingest /health (for plugins)     │
│  SQLite + FTS5(porter|trigram) + vector BLOBs, RRF(k=60) │
└──────────────────────────────────────────────────────────┘

The REST sidecar exists for multi-harness support: any agent harness with lifecycle hooks (Claude Code, Codex, Kimi Code, …) can wire the control plane as "hook event → curl localhost:8765" — only opencode v2 needs an in-process plugin, because its hooks live inside the JS runtime. See adapters/ for per-harness integrations (planned).

Design constraints: the plugin holds zero intelligence (retrieval, embedding and storage all live in the service); a dead service means silent skips, never a blocked session; no re-injection when the recalled set is unchanged (protects the prefix cache); the MCP server is fully usable without the plugin. The metric that matters is not "tokens saved" but recall@k of task-relevant context.

Related MCP server: rekal

Quick start

# 1. Install the service (uv or venv recommended)
pip install -e .                # BM25-only mode works out of the box
pip install -e ".[embed]"       # add the vector channel (builds llama-cpp-python)

# 2. (Optional) download the embedding model to enable the vector channel
bash scripts/download_model.sh  # prints the MEMORY_MODEL_PATH export line

# 3. Start the service (HTTP mode: MCP and REST share one port)
memory-mcp                      # http://127.0.0.1:8765/mcp
memory-mcp --stdio              # stdio mode (MCP tools only, no REST sidecar)

# 4. Register the MCP server in your harness — pick ONE of two modes

Mode A — shared HTTP service (recommended: one store shared across harnesses; the port is chosen at service startup, clients only reference it):

// opencode v2: ~/.config/opencode/opencode.jsonc
{ "mcp": { "memory": { "type": "remote", "url": "http://127.0.0.1:8765/mcp",
    "headers": { "Authorization": "Bearer {env:MEMORY_TOKEN}" } } } }  // headers only if MEMORY_TOKEN is set

// Claude Code: .mcp.json
{ "mcpServers": { "memory": { "type": "http",
    "url": "http://127.0.0.1:8765/mcp" } } }

Mode B — stdio local (each client spawns its own memory-mcp --stdio child; settings are passed via the config's env field — the only channel here; no port and no REST sidecar in this mode):

// opencode v2: opencode.jsonc  (field name `environment` per @opencode-ai/sdk types)
{ "mcp": { "memory": { "type": "local", "command": ["memory-mcp", "--stdio"],
    "environment": { "MEMORY_DB_PATH": "/data/memory/memory.db", "MEMORY_DEVICE": "cuda" } } } }

// Claude Code: .mcp.json
{ "mcpServers": { "memory": { "command": "memory-mcp", "args": ["--stdio"],
    "env": { "MEMORY_DB_PATH": "/data/memory/memory.db", "MEMORY_DEVICE": "cpu" } } } }

# Codex: ~/.codex/config.toml
[mcp_servers.memory]
command = "memory-mcp"
args = ["--stdio"]
env = { MEMORY_DB_PATH = "/data/memory/memory.db", MEMORY_DEVICE = "cpu" }

# 5. opencode v2: install the control-plane plugin (recall injection etc.)
cp opencode-plugin/memory.ts ~/.config/opencode/plugins/
# the plugin directory needs @opencode/plugin (types only):
#   cd ~/.config/opencode && npm install @opencode/plugin@^2.0.28
# — or declare it in opencode.jsonc's "plugin" array instead, optionally with
#   plugin options (see "Configuration" below):
#   "plugin": [["file:///abs/path/opencode-plugin", {"url": "http://127.0.0.1:9000"}]]

Configuration (environment variables)

Service:

Plugin:

Every MEMORY_PLUGIN_* variable has a plugin-options twin (url, token, limit, budget_chars, tool_capture, autostart, debug), so the plugin can also be configured right where it is declared in opencode.json — precedence: env var > plugin options > default:

{
  "plugin": [
    ["file:///path/to/semantic-memory-mcp/opencode-plugin",
     { "url": "http://127.0.0.1:9000", "limit": 8, "tool_capture": ["bash", "gis"] }]
  ]
}

The dev runtime (0.0.0-dev-*) passes the tuple's second element to server(ctx, options) (verified end-to-end); the stable 2.x runtime exposes the same table as ctx.options.

Random port & agent bootstrap

An agent that spawns the service itself can avoid port hunting and collisions entirely — MEMORY_PORT=0 lets the OS assign a free port, and the real address is published through two channels the moment the bind completes:

MEMORY_PORT=0 memory-mcp
# stdout (machine-readable, one line):
# MEMORY_MCP_LISTENING {"url":"http://127.0.0.1:41697","mcp_url":".../mcp","port":41697,"pid":...}
# and the same JSON in the discovery file: <db>.server.json (override: MEMORY_DISCOVERY_FILE)

So the bootstrap flow is: spawn → parse the stdout line or read the discovery file → point MEMORY_PLUGIN_URL / MCP registration at url. The file is retracted on graceful shutdown (SIGINT/SIGTERM); after a hard kill readers can still detect staleness via the embedded pid. stdio mode has no port and publishes nothing.

Indexing: near-real-time, no rebuild windows

• FTS (BM25): the FTS5 external-content tables are updated by triggers in the same transaction as the insert — a memory is keyword-searchable the moment memory_store returns.

• Vectors: the embedding is computed synchronously inside the write request and stored alongside the row. Retrieval is brute-force cosine over all vectors at query time — there is no ANN structure to build, lag, or ever rebuild.

• Cold start: the embedding model loads lazily on the first embed call (seconds, once per process). That first request blocks on it; nothing is dropped.

• Capture is at-most-once: hooks fire and forget (1.5s budget). Memories that arrive while the service is down are silently dropped — a blocked session is worse than a missed memory. Durability is best-effort, not exactly-once.

• Vector gap: entries written while the embedder is unconfigured stay BM25-searchable only; after installing the model, reconcile once with the memory-mcp backfill CLI (see Maintenance & lifecycle).

Storage layout

Everything lives in one SQLite file (MEMORY_DB_PATH, default ~/.local/share/memory-mcp/memory.db):

WAL sidecar files (-wal, -shm) exist at runtime; the GGUF model (~600 MB) is a separate file. Deleting the db is a clean full reset — there is no other state. Back up with sqlite3 memory.db ".backup '/path/to/backup.db'". Unlike ctx-semantic, the vector matrix is read from SQLite per query (the page cache carries this to ~50k entries) — zero cache-invalidation logic.

Sharing model

• Cross-session: the index is on disk; sessions are processes. Combined with the pre-compaction flush, even discarded context outlives its session.

• Cross-harness: opencode v2, Claude Code, Kimi Code and Codex all connect to the same service and therefore the same index — a memory stored by Claude Code is recallable from opencode. Switching harnesses loses nothing; there is no second copy to sync.

• project is a filter, not a partition: the index is global; the plugin filters by current project at injection time, but omitting project in memory_recall searches everything.

• Single user, single machine: loopback bind + optional Bearer token. No multi-tenancy.

• Attribution gap: source records the origin type (chat / compaction / tool) but not which harness wrote the entry — add a harness:<name> tag when that query need actually arises.

Choosing the storage location

MEMORY_DB_PATH is read at service startup (parent directories are auto-created). One db file per service instance — for hard db-level isolation (e.g. a classified project), run multiple instances on different ports and point MEMORY_PLUGIN_URL at the right one. Do NOT put the live db on NFS/SMB or synced folders (WAL locking semantics); use backups for cross-machine movement. Until a hard isolation need exists, prefer one instance + project filtering.

HTTP read/manage API + built-in web UI

The same process also serves a human-facing channel (distinct from the hook-facing /recall /ingest, same auth rule: Bearer when MEMORY_TOKEN is set):

Bulk-maintenance endpoints sit behind a double gate: they return 403 unless MEMORY_TOKEN is configured (on a default no-token loopback install the HTTP surface is open to every local process — including the agent's shell — so there the capability stays CLI-only), and 401 without a valid Bearer token. The browse page exposes both as buttons (confirm → report → stats refresh).

Open http://127.0.0.1:8765/ for stats overview, RRF search, kind/project filtering, single-entry forgetting, and (on token-secured installs) one-click maintenance. There is still no bulk editing; observability is what makes "relevance is never lost" believable.

Injection telemetry. Recall ≠ injection — only the injecting side knows a block actually entered the context, so the opencode plugin reports each real injection to POST /inject/report (fire-and-forget): harness, hook, trigger query, memory uids, block size. The browse page shows the supply-side picture: total/7-day injection counts, characters (with a rough ÷4 token estimate), per-harness/per-hook distribution, the Top10 most-injected memories, and the 20 most recent injections. The shell adapters (Claude Code / Kimi Code) report the same way — recall.sh fires the report after emitting a block, tagged via MEMORY_HARNESS / MEMORY_HOOK. This measures what was supplied — whether the model used it is not measurable and deliberately not faked. Telemetry rows expire after MEMORY_TTL_INJECTION_DAYS (default 90).

Context hygiene. Memory content is untrusted data that ends up inside a prompt. Every injected block carries an explicit "reference data, not instructions" notice, and any <semantic-memory> boundary tags inside the content itself are stripped so a stored entry cannot break out of its wrapper. On the write side, tool-output capture skips artifacts matching classic prompt-injection patterns ("ignore all previous instructions", "忽略…指令", <system>, …) so a poisoned tool result cannot re-enter later sessions through the memory store.

Maintenance & lifecycle

Retention policy is configuration, not LLM discretion. Bulk destructive operations are deliberately not exposed as MCP tools — they live where prompt injection cannot reach: the startup hook and the CLI. The agent can only memory_forget (single entry, user-initiated) and memory_stats (observe and recommend).

• Ingest dedup (in the write path, nobody "calls" it): exact content-hash match (global) or vector near-duplicate (same project, cosine ≥ MEMORY_DEDUP_SIM) refreshes the existing entry's timestamp instead of inserting — recurrence is treated as "proven relevant again". This caps the growth slope of compaction-flush sources, which love re-storing near-identical content.

• Retention tiers: preference / decision / fact are permanent; artifact (tool captures) expires after MEMORY_TTL_ARTIFACT_DAYS; injection telemetry expires after MEMORY_TTL_INJECTION_DAYS.

• memory-mcp purge: hard-deletes soft-deleted entries, applies the TTL, then VACUUM (only when something was actually removed). Same routine runs at every service start (MEMORY_MAINTENANCE_ON_START=1) — restart frequency is enough for day-granularity retention; no cron, no background threads.

• memory-mcp backfill: embeds every live entry that has no vector yet (one-shot reconcile after installing the model). Manual because it is heavy.

Vector channel (same vectors as ctx-semantic)

The embedding layer mirrors ctx-semantic item by item — same model file, same llama-cpp-python pin (0.3.35), same output:

• Model: Qwen3-Embedding-0.6B Q8_0 GGUF, dim 1024, pooling LAST (explicit)

• Asymmetric embedding: queries get an Instruct: ...\nQuery: {text} prefix, documents are embedded raw

• n_ctx=8192, sub-batches of 32, float32 rows L2-normalized (1e-12 guard)

• Vector search: brute-force cosine (sub-millisecond up to ~50k entries, no ANN needed)

Device policy (deliberately inverted vs. ctx-semantic): MEMORY_DEVICE=cuda is the default, cpu is opt-in. The GPU path falls back deterministically: load → probe embed → nvidia-smi PID verification (detects silent CPU fallback) → any failure rebuilds on CPU with a log line; a runtime GPU embed failure (e.g. OOM) rebuilds on CPU and retries once. On machines without NVIDIA, set MEMORY_DEVICE=cpu to skip the GPU attempt. memory_stats / /health report the actual device in use.

Ranking post-processing (measured, not guessed)

RRF encodes only rank, not relevance strength — "nearest among the irrelevant" still earns a score. Two post-fusion steps, both on by default:

• Cosine floor (MEMORY_RECALL_MIN_COSINE=0.45): vector-only hits below the floor are dropped; BM25 hits always pass (a token match is evidence by itself). The floor is measured on the real model: relevant hits land ~0.68, unrelated ~0.28-0.31.

• Telemetry feedback (MEMORY_INJECT_BOOST=0.15): a memory that was actually injected before scores × (1 + α·ln(1+injections)). The log dampens rich-get-richer; 2 prior injections flip an adjacent rank pair.

Multilingual full-text search

• The porter channel (unicode61 + English stemming) indexes every script with word boundaries: English, French, German, Spanish, Russian, Greek, Arabic, …

• The trigram channel (≥3-char substrings) covers scripts without word boundaries: Chinese, Japanese, Korean, Thai, Lao, Khmer, Burmese — and provides substring fallback for any language (even German compounds).

Known gap: 2-character CJK words (e.g. 坐标, 投影) fall below the trigram minimum. The vector channel backstops this (Qwen3-Embedding is multilingual, 100+ languages); if BM25-only mode ever shows a real miss, the fix is an application-level bigram channel (~15 lines, zero dependencies) — see the roadmap.

Testing

pip install -e ".[dev]"
pytest            # 43 tests: store / dual-channel recall / soft delete / RRF math /
                  # CJK substring / query-prefix contract / fault tolerance /
                  # lifecycle dedup & TTL / migration / REST & /api routes /
                  # bulk double-gate auth / MCP tool layer /
                  # random-port serve & service discovery /
                  # injection telemetry (report/aggregation/TTL/auth) /
                  # ranking post-processing (cosine floor, telemetry boost) /
                  # real-model vector E2E (opt-in via MEMORY_MODEL_PATH)

cd opencode-plugin
npm install       # then:
npm run typecheck # tsc --strict against @opencode/plugin 2.0.28 types
npm test          # build + 37 mock-ctx hook assertions against a live service
                  # (stable line: inject/dedup/remember/compaction/tool-capture/
                  #  telemetry/capture-hygiene/tag-sanitizing; legacy line: v1
                  #  hook map, synthetic parts, compacting reset, telemetry;
                  #  options: env wins, gap-filling)

The plugin is additionally verified against the real opencode dev loader (0.0.0-dev-202610130450): it loads the dual-form export, selects the legacy line, and resolves the workspace as project — no load errors.

Roadmap

• Phase 1: MCP tools + hybrid retrieval + REST sidecar (current state)

• Phase 2: opencode v2 plugin, four hook families (inject / pre-compaction flush / explicit remember / tool whitelist)

• Phase 3a: multi-harness adapters (adapters/ — Claude Code, Kimi Code, Codex template)

• Phase 3b: lifecycle maintenance — ingest dedup, TTL/purge/VACUUM, vector backfill (startup hook + CLI, deliberately off the MCP tool surface)

• Phase 3c: semantic-level auto-capture for chat.message, bigram channel for 2-char CJK terms (trigger: a real recall miss in BM25-only mode), cross-project memory graph, memory archive (cold file)

Acknowledgments

Architecture inspired by Basic Memory (Markdown + SQLite FTS), ctx-semantic (BM25 + Qwen3-Embedding + RRF), and context-mode (hook-interception paradigm).