mcp-memory-search

An MCP server that searches a folder of Markdown notes and returns the passages most relevant to a query. It combines vector search (meaning) with SQLite FTS5 (exact terms), and everything runs locally on CPU.

I built this because grepping my own notes stopped scaling, and handing the whole pile to an agent blew past the context window. Now Claude Code (or any MCP client) can just ask for the relevant bits.

Why local

Embeddings run through fastembed, which is ONNX, so no PyTorch and no GPU. The index is one SQLite file (sqlite-vec plus an FTS5 table). There's no hosted embedding API in the loop, so no quota and no network calls. The transport is stdio only, so there's nothing listening on a port.

Related MCP server: recall-mcp

Tools

Setup

git clone https://github.com/<you>/mcp-memory-search
cd mcp-memory-search
python -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt

cp config.example.json config.json     # edit "roots" to point at your notes
python ingest.py                       # first run downloads a ~220 MB model
python test_client.py                  # optional smoke test

config.json:

{
  "roots": ["~/Documents/notes", "./notes"],
  "extensions": [".md", ".txt"],
  "recursive": true,
  "max_file_kb": 120,
  "include_docx": false
}

MEMORY_SEARCH_ROOTS="/path/a:/path/b" overrides roots if you'd rather use an env var.

Adding it to an MCP client

For Claude Code (~/.claude.json) or Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "memory-search": {
      "command": "/abs/path/mcp-memory-search/.venv/bin/python",
      "args": ["/abs/path/mcp-memory-search/server.py"]
    }
  }
}

Keeping it current

python ingest.py skips any file whose content hasn't changed (mtime + sha1), so a re-run over a thousand unchanged notes finishes in a fraction of a second. Run it from cron, a systemd timer, or a file watcher. The Markdown is the real data; memory.db is just an index and you can delete it and rebuild.

How it works

ingest.py splits each file into chunks along its headings (target ~1200 chars, 150 overlap) and prepends a title taken from the frontmatter name: and description: fields. Each chunk goes into sqlite-vec as a 384-dim vector and into the FTS5 table. On a query, memory_search runs a vector KNN and a BM25 search, then merges the two rank lists with Reciprocal Rank Fusion. Having the FTS half matters: the small embedding model is hit-or-miss on codenames and acronyms, and BM25 covers those. Results are capped at two chunks per file so one long note can't crowd everything else out.

The model is paraphrase-multilingual-MiniLM-L12-v2. It loads on the first query and unloads itself after ten minutes idle, so it isn't holding RAM when nothing is using it.

Limitations

• Markdown and plain text only (.docx if you enable it). PDFs need to be converted first.

• The small model is a speed/size tradeoff. The FTS side of the hybrid is there partly to make up for it on exact lookups.

• One index per checkout, in memory.db next to the code.

License

MIT.