memory-core
by davebco
README.md
# memory-core
A self-hosted, database-backed long-term memory for AI agents.
Your agent's memories live in Postgres as small Markdown notes. memory-core serves them over
**REST** and **MCP** (Model Context Protocol) with hybrid search, per-client tokens, version
history, and a secret scanner on every write. It was built to give Claude Code one shared memory
across several machines, but any MCP client or HTTP caller can use it.
> **Status:** a personal project that runs one household's agent memory every day. It is small
> (about 2,000 lines of Python), tested, and opinionated. It is shared as-is: expect to read the
> code, and expect rough edges outside the author's setup (Docker Swarm, Windows clients).
## What it does
- **Hybrid search.** BM25 keyword search ([pg_textsearch](https://github.com/timescale/pg_textsearch))
and vector search ([pgvector](https://github.com/pgvector/pgvector)) are merged with reciprocal
rank fusion, then re-scored by a cross-encoder. Exact strings such as IPs and model ids work as
well as plain-language questions.
- **An empty answer when there is no answer.** A score floor drops weak hits, so an off-topic
question returns nothing instead of five plausible-looking wrong notes.
- **Local models.** Embedding and rerank models are baked into the image and run on CPU. No
memory text leaves the host.
- **Scoped bearer tokens.** One token per client, each revocable alone: `read`, `suggest` (writes
are queued for review), `write`, `admin`. Only a SHA-256 hash of each token is stored.
- **Owner isolation.** Postgres row-level security scopes every query to the token's owner.
- **Secret scanning.** A write that contains an API key, private key, JWT or password-like
assignment is rejected. Store a pointer (`secret:NAME`) instead of the value.
- **Versioning.** Every change is kept in a history table. Writes accept `expected_version`
for optimistic locking, and any version can be restored.
- **Markdown in, Markdown out.** Import a folder of `.md` files with YAML front matter, and export
back to the same format on a schedule.
- **Claude Code hooks** (optional): automatic recall on each prompt, a read-only local mirror of
the memories, and an offline outbox for when the server is unreachable.
## How it fits together
```
MCP client / REST caller
│ Authorization: Bearer mc_<name>_<random>
▼
api (FastAPI + FastMCP, port 8020)
│ embed query → BM25 + vector → RRF → cross-encoder rerank → score floor
▼
db (Postgres 17 + pgvector + pg_textsearch, row-level security)
```
| Service in `docker/stack.yml` | What it does |
|---|---|
| `db` | Postgres 17 with pgvector and pg_textsearch. Data lives in the volume `memcore_pgdata`. |
| `migrate` | One-shot job: applies migrations, creates or rotates the `memcore_app` role, then exits. |
| `api` | uvicorn on port 8020. REST under `/api`, MCP at `/mcp/`, health at `/healthz`. |
| `exporter` | Exports one owner's memories to Markdown every hour (optional). |
| `maintenance` | Daily `VACUUM` and `REINDEX` of the search indexes. Without it, recall slowly drops as notes are edited. |
## Quick start
You need Docker with Swarm mode on (`docker swarm init` is enough for a single host). The stack
uses Swarm secrets, so no password ever sits in a file or an environment variable.
**1. Create the secrets.**
```sh
python -c "import secrets;print(secrets.token_urlsafe(32))" | docker secret create memcore_db_admin_password -
python -c "import secrets;print(secrets.token_urlsafe(32))" | docker secret create memcore_db_app_password -
docker secret create memcore_tls_cert fullchain.pem # leaf certificate first, then intermediates
docker secret create memcore_tls_key privkey.pem
```
No TLS certificate yet? Remove the two `MC_TLS_*` lines and the two TLS secrets from the `api`
service and it serves plain HTTP. Only do that behind a TLS-terminating proxy or on a network you
fully trust: the bearer tokens travel in a header.
**2. Deploy.**
Edit `docker/stack.yml` first: set the exporter's `--owner` and its bind-mount path (the folder
must exist), or delete the `exporter` service.
```sh
docker stack deploy -c docker/stack.yml memory-core
curl https://<host>:8020/healthz # -> {"status":"ok"}
```
The images are built by `.github/workflows/publish.yml`. The app image is large because the
models are baked in, and the first start takes a moment while they load.
**3. Mint a token.** The token is printed once and cannot be shown again.
```sh
docker exec $(docker ps -q -f name=memory-core_api) \
python -m app.cli token create --name my-laptop --owner alice --scope write
```
`token list` and `token revoke --name <name>` work the same way. `--owner` is the tenant: tokens
with the same owner share one set of memories, and can never see another owner's.
**4. Connect a client.** For Claude Code:
```sh
claude mcp add --transport http memory-core https://<host>:8020/mcp/ \
--header "Authorization: Bearer <token>"
```
Keep the trailing slash on `/mcp/`.
## MCP tools
| Tool | Scope | What it does |
|---|---|---|
| `memory_search` | read | Hybrid search. Returns the best-matching passages with their memory keys. |
| `memory_get` | read | One full memory, by key or by name. |
| `memory_list` | read | All memories; filter by `type`, optionally include archived ones. |
| `memory_history` | read | The version list for one memory. |
| `memory_write` | suggest | Create or replace a memory. With a `suggest` token the write is queued for review. |
| `memory_edit` | write | Replace one exact string in a memory's body. It must match exactly once. |
## REST API
Every call needs `Authorization: Bearer <token>`, except `/healthz`.
| Method and path | Scope | Notes |
|---|---|---|
| `POST /api/search` | read | Body: `{"query": "...", "k": 5, "type": null, "include_archived": false}` |
| `GET /api/list` | read | Query: `type`, `include_archived` |
| `GET /api/memory/{key}` | read | 404 if the key does not exist |
| `POST /api/memory` | suggest | Create or replace. Optional `expected_version`; a stale one returns 409 |
| `POST /api/memory/{key}/edit` | write | Body: `{"old_str": "...", "new_str": "..."}` |
| `GET /api/memory/{key}/history` | read | Version list |
| `GET /api/memory/{key}/history/{version}` | read | That version's full snapshot |
| `POST /api/memory/{key}/revert` | write | Body: `{"version": N}` |
A rejected secret returns HTTP 422 with the kind of secret found, never the value.
## A memory
A memory is a key, a name, a short description, an optional type, and a Markdown body. As a file:
```markdown
---
name: Guest wifi
description: network name and where the password lives
metadata:
type: reference
---
## Current state
The guest network is `house-guest`. Password: secret:GUEST_WIFI (in the password manager).
```
The key is the filename stem (`reference_guest_wifi`). Bodies are split into chunks by heading,
and each chunk is embedded together with the memory's name and description.
## Configuration
Server settings are environment variables on the `api` and `migrate` services.
| Variable | Default | Meaning |
|---|---|---|
| `MC_DB_HOST`, `MC_DB_PORT`, `MC_DB_NAME` | `localhost`, `5432`, `memcore` | Where Postgres is |
| `MC_EMBED_MODEL`, `MC_EMBED_DIM` | `BAAI/bge-base-en-v1.5`, `768` | Embedding model. The api refuses to start if these differ from what the database was created with |
| `MC_RERANK_MODEL` | `BAAI/bge-reranker-base` | Cross-encoder. Empty turns rerank off |
| `MC_CANDIDATE_K` | `50` | Candidates fetched from each of BM25 and vector search |
| `MC_RERANK_TOP` | `20` | How many fused candidates are reranked |
| `MC_RERANK_MAX_LENGTH` | model default | Token cap for the cross-encoder input |
| `MC_MIN_RERANK` | `0.10` | Score floor when rerank is on |
| `MC_MIN_COSINE` | `0.56` | Score floor when rerank is off |
| `MC_TLS_CERT_FILE`, `MC_TLS_KEY_FILE` | unset | Set both to serve HTTPS |
The score floors depend on your data and your rerank model. The value in `docker/stack.yml` was
tuned on the author's memories; tune yours with the eval tool below.
## Import and export
Import a folder of Markdown memories through the API. Each write is authenticated, secret-scanned
and versioned, and running it again changes nothing:
```sh
export MC_IMPORT_TOKEN=<a write token>
python -m app.importer --dir ./memory --url https://<host>:8020
```
A `rejected` entry means the secret scanner found a value. Replace it with a pointer. If it is a
false positive, add it as a `SAFE` case in `tests/test_secret_scan.py` and fix the pattern.
Export runs as the `exporter` service, or by hand: `python -m app.cli export --owner alice --out ./export`.
## Claude Code integration (optional)
The files in `hooks/` are Claude Code hooks. They use only the Python standard library and read
their settings from `~/.memory-core/.env` (override the path with `MC_ENV_FILE`).
| Client variable | Meaning |
|---|---|
| `MC_URL` | Base URL of the server, e.g. `https://memory.example.internal` |
| `MC_CLAUDE_CODE_TOKEN` | A `read` token for the recall hook |
| `MC_SYNC_TOKEN` | A `write` token for sync. Use a different token name on each machine |
| `MC_MASTER` | `files` (default) or `db`. Which side owns the memories; see below |
| `MC_MEMORY_DIR` | The local memory folder. Default `~/.memory-core/memory` |
| `MC_LOG_DIR` | Where hooks write logs and the outbox. Default `~/.memory-core/log` |
| `MC_OUTBOX` | `off` on a machine that runs no down-sync |
| Hook | Event | What it does |
|---|---|---|
| `hooks/memory_recall.py` | UserPromptSubmit | Searches for the prompt and hands Claude the top 3 hits. Fails silently |
| `hooks/memory_sync.py` | PostToolUse | `files` mode: pushes each edited memory file to the server |
| `hooks/memory_guard.py` | PreToolUse | `db` mode: blocks direct edits to the mirror and points at the MCP tools |
| `hooks/memory_after_write.py` | PostToolUse | `db` mode: refreshes the mirror after a write; offers the outbox if the server is down |
| `hooks/outbox.py` | — | Queues a save while the server is unreachable; replayed later with a version check |
**Two modes.** In `files` mode the Markdown folder is the master and `python -m app.sweep` keeps
the server in step with it. In `db` mode the database is the master and `python -m app.downsync`
writes a read-only Markdown mirror, so plain `grep` still works when the server is down.
`scripts/register-sweep-task.ps1` and `scripts/register-downsync-task.ps1` register these as
Windows scheduled tasks; on Linux or macOS use cron or a systemd timer.
`scripts/mint_to_env.py` mints a token over SSH and writes it straight into the env file without
printing it. `scripts/health_line.py` prints a one-line health summary for a daily report.
## Measuring search quality
`eval/queries.json` is a small **sample**. Replace it with real questions about your own memories:
plain-language questions, exact identifiers, and "noise" questions that should return nothing.
```sh
export MC_EVAL_TOKEN=<a read token>
python eval/run_eval.py --url https://<host>:8020 --label baseline
python scripts/bench.py https://<host>:8020 # p50 / p95 search latency
```
`run_eval.py` reports Recall@1/3/5 and MRR for each set and how many noise queries wrongly
returned a hit. Use it whenever you change a model or a score floor.
## Security notes
- **Built for a private network.** Run it on a LAN or VPN, or behind a reverse proxy. It has no
rate limiting and has not been hardened for the open internet.
- **DNS-rebinding protection is off** in the MCP transport (`app/api.py`), because every call is
bearer-authenticated and the service is meant to be reached by several hostnames. Turn it back
on if you expose the server more widely.
- **Tokens are shown once.** Only their hashes are stored. Keep them out of repositories and
tracked settings files, and give each client its own so one can be revoked alone.
- **The secret scanner is a safety net, not a guarantee.** It matches known patterns.
- `/healthz` is unauthenticated and returns only `{"status":"ok"}`.
Found a security problem? Please open a GitHub issue. Leave out anything sensitive, and ask for
a private channel if you need one.
## Development
Tests need a throwaway Postgres with both extensions. `docker/dev-db.yml` starts one on port 5433:
```sh
MC_DEV_DB_PASSWORD=<pick one> docker stack deploy -c docker/dev-db.yml memory-core-dev
pip install -r requirements-dev.txt
export MC_TEST_ADMIN_URL=postgresql://postgres:<that password>@localhost:5433/postgres
python -m pytest
```
Each test run creates its own database and drops it afterwards. Tests use a fake 8-dimension
embedder; set `RUN_REAL_MODELS=1` to also run the tests that download the real models.
## License
[MIT](LICENSE)
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo recent community issues