agent-memory MCP server

The idea: your notes are the database. The index is disposable. The embedding model is loaded on demand and released, not a resident service.

Why

Agent memory usually fails in one of three ways:

How it compares

Related MCP server: obsidian-brain

Highlights

How it works

flowchart LR
    Q([query]) --> F["FTS5 keyword search<br/>BM25 + IDF coverage"]
    F --> C{"confidence ≥ 0.25?"}
    C -- yes --> R["RRF fusion"]
    C -- no --> M["load model → embed query"]
    M --> V["vector search"]
    V --> R
    R --> S["token-cheap summary"]
    D[("SQLite<br/>docs · vectors · FTS5")] -.-> F
    D -.-> V

1. Keyword stream. Query tokens are stopword-filtered; tokens of 4+ characters get prefix matching. BM25 ranks the candidates.

2. Confidence gate. IDF-weighted coverage measures how much of the query the best document actually contains. At ≥ 0.25 the keyword stream is trusted and the model is never opened. --semantic bypasses the gate.

3. Semantic stream. Notes are embedded once and cached as raw float32 blobs; the query is embedded with the model's instruction prefix.

4. Fusion. RRF (k = 60) merges both rankings without comparable scores.

That is why a typical lookup returns in ~0.01–0.2 s and costs no VRAM.

Quick start

git clone https://github.com/aqa03009-ux/agent-memory.git
cd agent-memory

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

export AGENT_MEMORY_VAULT="$PWD/examples/vault"
./memory.py reindex
./memory.py search "why is the model lazy loaded"
./memory.py status

The first reindex downloads the embedding model (~1.2 GB) and embeds the notes. Later runs only touch changed files.

If you have an accelerator, install its torch build before pip install -r requirements.txt (see the GPU / ROCm notes below). The default PyPI wheel can be a multi-gigabyte CUDA download even on CPU-only machines.

The default device is CPU on purpose. On the machine this was extracted from, the ROCm path with bf16 produced NaN vectors; float32 was clean but cost ~2.5 GB of VRAM on every invocation. CPU float32 is the safe default for a 0.6B model.

AGENT_MEMORY_DEVICE=auto ./memory.py reindex   # try CUDA/ROCm, fall back to CPU
AGENT_MEMORY_DEVICE=cpu  ./memory.py reindex   # force CPU (default)

For ROCm, install the matching torch build first, then pip install -r requirements.txt.

Usage

Vault layout

vault/
├── 00-index.md          # entry point / map
├── profile.md           # canonical facts about the user
├── setups/              # how working things are configured
├── decisions/           # why something was chosen
├── projects/            # one note per project
├── research/            # deep-dive reports
├── sessions/            # session handoffs (open/claim queue)
├── daily/               # quick notes
├── assets/              # media (optional)
├── relations.md         # generated link graph
├── .memoryignore        # paths excluded from the index
└── .vectors/memory.db   # derived index (git-ignored)

Copy .memoryignore.example to .memoryignore when you want to exclude raw corpora (transcripts, logs) from retrieval.

Every note is plain Markdown:

# Local search setup

tags: #setup #search #local

Details here. Link related notes with [[decisions/example-decision]].

MCP server

memory.py mcp speaks JSON-RPC 2.0 over stdin/stdout and exposes eight tools:

{
  "mcp": {
    "memory": {
      "type": "local",
      "command": [
        "/absolute/path/to/.venv/bin/python",
        "/absolute/path/to/memory.py",
        "mcp"
      ],
      "environment": {
        "AGENT_MEMORY_VAULT": "/home/you/memory"
      },
      "enabled": true
    }
  }
}

{
  "mcpServers": {
    "memory": {
      "command": "/absolute/path/to/.venv/bin/python",
      "args": ["/absolute/path/to/memory.py", "mcp"],
      "env": { "AGENT_MEMORY_VAULT": "/home/you/memory" }
    }
  }
}

The model is loaded inside the tool call and unloaded when the call ends, so an idle MCP server holds no model in memory. Set AGENT_MEMORY_MCP_KEEP_WARM=1 if you prefer to keep it warm between calls.

Session handoffs

memory.py open lists handoff notes that are still unclaimed. memory.py claim rewrites the note's status: line to claimed (<timestamp>), so two agents never pick up the same task. Notes using the original Turkish marker (durum: devralındı or devralindi) are also recognized.

# Finish the index migration

status: open
recorded: 2026-01-01 10:00
session: abc123

## What the user asked
...

## Where we stopped
...

Configuration

Using a different model? Set AGENT_MEMORY_MODEL and clear AGENT_MEMORY_QUERY_INSTRUCT if it does not use query instructions. Changing the model or dimension automatically forces a full reindex, because old vectors live in a different space.

Design notes

• Incremental by mtime + identity. Reindex compares file mtime and checks that a vector exists; it also stores model/dim metadata and rebuilds when they change.

• Derived data is disposable. .vectors/ is git-ignored. Delete it and reindex; the Markdown never depends on the database.

• Honest failure. If the model is unavailable, the index keeps working in keyword mode. If a vector is stale, it is dropped rather than silently kept.

• One reindex at a time. A flock on reindex.lock prevents plugin-triggered background runs from stepping on each other.

• All vault paths are contained. show, add, and claim resolve their arguments and reject absolute paths or .. traversal, so a prompt-injected note cannot turn an MCP tool into an arbitrary file read or write.

• Long notes are indexed in full. Embeddings cover the first 4,000 characters, which is deliberate for note-sized documents; for books, split them into notes.

Limitations

• Single-user, local, no authentication. Do not expose the MCP server over a network.

• FTS5 requires a Python build of SQLite with FTS5 enabled (most Linux distros are fine).

• Media embeddings need a multimodal model (e.g. google/embeddinggemma-2) plus pillow, soundfile, librosa, and ffmpeg. The text path does not. Text and media share one vector space, so switching models requires a full reindex.

• Tested on Linux with Python 3.11 and 3.12 (CI runs 3.12), using Qwen3-Embedding-0.6B. The design is model-agnostic, but the MRL and instruction defaults are Qwen-specific.

License

MIT — see LICENSE.