[简体中文](README.zh.md)

# Memory Vault

A **cross-session persistent memory plugin** for coding agents: save the experiences, decisions, preferences and pitfalls distilled from conversations to your local machine, and recall them on demand in future sessions.

- **Storage**: SQLite local database (single source of truth); records, tags and the inverted word table all persisted to disk
- **Retrieval**: dual-channel fused ranking of keywords (BM25) and semantic vectors (USearch, when available), with time decay support
- **Curation**: automatic deduplication on write (exact body dedup + semantic approximate dedup, with `merge`/`skip` strategies); `tidy` collapses similar records into summaries in one step to prevent memory bloat
- **UI**: built-in web viewing interface (pure standard library), browse, search, add, delete, tidy
- **Compat**: generic markdown memory format import/export, supporting frontmatter and splitting by second-level headings
- **Zero-dependency runnable**: the default local hash embedder works offline and is deterministically consistent across sessions; optionally plug in sentence-transformers or any OpenAI-compatible embedding API; vector retrieval auto-degrades (USearch → numpy → pure Python)

## Quick Start

No dependencies to install; Python ≥ 3.9 is enough:

```bash
# 存入一条记忆
python -m memory_vault put "项目架构" "后端采用微服务，服务间通过消息队列通信"

# 混合检索
python -m memory_vault ask "微服务架构"

# 从 markdown 目录批量导入
python -m memory_vault import ./notes --split

# 压缩整理
python -m memory_vault tidy

# 启动 Web 查看界面
python -m memory_vault serve
# 浏览器打开 http://127.0.0.1:8988

# 以 MCP 服务运行（供插件化 harness 加载）
python -m memory_vault mcp
```

Data is stored by default in `~/.memory-vault/` (`vault.sqlite3` and an optional `config.json`). You can also point it elsewhere with `--vault <dir>` or the `VAULT_DIR` environment variable.

## Integrating with dsh (plugin harness)

### Installing in DSH

```bash
dsh plugin --profile demo add github:JohnXu22786/memory-vault
```

The repo also ships a `dsh.bundle` (`package.json` + `cordis.patch.yml` + `index.js`).
Installing it lays down a Cordis plugin row whose Node bridge drives the Python CLI
(everything loads from the same package directory), surfacing the CLI commands as
dsh tools `vault_put` / `vault_ask` / `vault_take` / `vault_list` / `vault_drop` /
`vault_tidy` / `vault_stats` / `vault_ingest`. No npm dependencies are required at
load time; Python ≥ 3.9 must be on `PATH` (override the interpreter with the
`DSH_MV_PYTHON` environment variable). If Python is missing or the package can't be
imported, the bridge logs a clear error at startup instead of crashing. For
full-fidelity write options (tags / weight / source), use the MCP server below.

The plugin root contains `manifest.json`; the harness loads it per the following conventions:

| Interface | How it starts | Purpose |
| ---- | -------- | ---- |
| **MCP tools** (recommended) | `python -m memory_vault mcp` (stdio transport) | exposes 8 tools such as `vault_put` / `vault_ask` to the agent for reading/writing memory in-session |
| **CLI** | `python -m memory_vault <command>` | scripting, scheduled tidy, batch import/export |
| **Web** | `python -m memory_vault serve` | human browsing and maintenance interface |
| **Skill** | read `SKILL.md` | instruction text guiding the agent on when to write and how to retrieve |

Typical harness config sketch (MCP style):

```jsonc
{
  "mcpServers": {
    "memory-vault": {
      "command": ["python", "-m", "memory_vault", "mcp"],
      "env": { "VAULT_DIR": "~/.memory-vault" }
    }
  }
}
```

After the harness launches the process, it sends the `initialize` handshake, then discovers tools via `tools/list` and calls them via `tools/call`. The protocol is newline-delimited JSON-RPC 2.0 over stdio (MCP standard transport), with no third-party dependencies.

### MCP tools at a glance

| Tool | Description |
| ---- | ---- |
| `vault_put` | store a record (auto-dedup; returns `action: new/merged/skipped`) |
| `vault_ask` | hybrid retrieval, returns `id/title/score/kw/sem/updated_at` |
| `vault_take` | fetch full content by id |
| `vault_list` | list recent records |
| `vault_drop` | delete by id |
| `vault_tidy` | tidy up (collapse similar records into summaries) |
| `vault_stats` | storage stats (count, vector backend, embedding config) |
| `vault_ingest` | batch import from markdown files/directories |

## Usage Tips

- **What to store**: project decisions and their reasons, pitfalls hit, user preferences, common commands and conventions, experiment conclusions
- **When to retrieve**: at the start of a new session, when a task arrives with insufficient context, when a historical topic comes up
- **Don't over-store**: stable engineering rules belong in AGENTS.md-style documents; memory is meant to hold "context that grew out of real work"

## Configuration

The config file defaults to `<data-dir>/config.json` (JSON); `VAULT_*` environment variables can override it; command-line arguments take the highest precedence. A full example is in `config.example.json`.

| Section | Key | Default | Description |
| -- | -- | ---- | ---- |
| database | dir | `~/.memory-vault` | data directory |
| embedding | provider | `local` | `local` (built-in hash) / `sentence` (sentence-transformers) / `api` (OpenAI-compatible) |
| embedding | model | per provider | sentence defaults to `all-MiniLM-L6-v2`; api defaults to `text-embedding-3-small` |
| embedding | dims | `128` | local embedding dimensions |
| embedding | api_url / api_key / api_model | empty | api provider endpoint; keys support `env://VAR`, `file:///path` forms |
| search | keyword_weight / semantic_weight | `0.4` / `0.6` | dual-channel fusion weights (auto-clamped to 0~1) |
| search | recency_days | `180` | time-decay half-life (days), `0` disables |
| search | top_k | `8` | default number of results |
| curation | dedup_threshold | `0.92` | write-time dedup similarity threshold |
| curation | dedup_mode | `merge` | `merge` into the existing record / `skip` keeping the original |
| curation | cluster_threshold | `0.78` | tidy clustering threshold |
| curation | min_cluster | `2` | minimum cluster size |
| curation | digest_member_chars | `600` | characters retained per member record in the summary |
| web | host / port | `127.0.0.1` / `8988` | web interface listen address |
| web | token | empty | when set, all `/api/*` requests must carry `Authorization: Bearer <token>` (the UI prompts and remembers it on first request; `?token=` query param also supported) |

Environment variables: `VAULT_DIR`, `VAULT_CONFIG`, `VAULT_EMBED_PROVIDER`, `VAULT_EMBED_MODEL`, `VAULT_EMBED_DIMS`, `VAULT_EMBED_API_URL`, `VAULT_EMBED_API_KEY`, `VAULT_EMBED_API_MODEL`, `VAULT_KW_WEIGHT`, `VAULT_SEM_WEIGHT`, `VAULT_RECENCY_DAYS`, `VAULT_TOP_K`, `VAULT_DEDUP_THRESHOLD`, `VAULT_DEDUP_MODE`, `VAULT_CLUSTER_THRESHOLD`, `VAULT_MIN_CLUSTER`, `VAULT_DIGEST_MEMBER_CHARS`, `VAULT_WEB_HOST`, `VAULT_WEB_PORT`, `VAULT_WEB_TOKEN`.

## Embedding Providers

| provider | prerequisite | traits |
| -------- | -------- | ---- |
| `local` (default) | none | zero-dependency, offline, deterministic; limited semantic ability, good for getting started and testing |
| `sentence` | `pip install sentence-transformers` | real local semantic model, best results, fully offline |
| `api` | an accessible OpenAI-compatible endpoint | configure `api_url` + `api_key` (+ `api_model`), e.g. `https://api.openai.com/v1`, a local Ollama-compatible gateway, etc. |

> If changing the embedding config changes the dimensions, the plugin automatically re-embeds existing records on next use.

## Vector Retrieval Backend

Prefers [USearch](https://github.com/unum-cloud/usearch) (`pip install usearch`) approximate nearest neighbor; if not installed it auto-degrades to a numpy exact scan, then to a pure Python scan. SQLite is always the single source of truth; the in-memory index is rebuilt from the database at every startup.

## Markdown Compatibility

- **Import**: `import <file-or-directory> [--split]`. Recognizes YAML-style frontmatter (`title/tags/weight/created/updated`), `# H1 heading` (used as the title and stripped from the body; CRLF line endings supported); `--split` splits into multiple records by `## H2 headings`
- **Export**: `export <dir>`, one `.md` file per record (frontmatter contains id/time/tags), re-importable
- **Edge cases**: record ids are always generated by the system; the `id` in frontmatter is only exported as information and ignored on import; tags must not contain commas; leading/trailing whitespace in bodies is trimmed on export
- Designed to interoperate with existing markdown note libraries

## CLI Reference

```
python -m memory_vault init                    # 初始化数据目录
python -m memory_vault put "标题" "正文"        # 存入（正文可省略，此时读标准输入）
echo "正文" | python -m memory_vault put "标题"
python -m memory_vault ask "查询词" -k 5        # 混合检索
python -m memory_vault get <id>                 # 查看单条
python -m memory_vault list -n 20               # 最近列表
python -m memory_vault drop <id>                # 删除
python -m memory_vault tidy                     # 压缩整理
python -m memory_vault import ./notes --split   # 导入 markdown
python -m memory_vault export ./backup          # 导出 markdown
python -m memory_vault info                     # 统计
python -m memory_vault serve                    # Web 界面
python -m memory_vault mcp                      # MCP 服务
```

All commands support `--vault <dir>`, `--config <file>` and `--json` (machine-readable output). `--vault`/`--config` are global options and must come before the subcommand (e.g. `python -m memory_vault --vault ~/mv put ...`).

> **Safety note**: the web interface listens only on `127.0.0.1` by default. If you need to bind to a non-loopback address (e.g. `0.0.0.0`), make sure to also set `web.token`; the UI has built-in cross-origin write protection (enforced JSON Content-Type + Origin check) and request timeout/concurrency limits.

## Web API

| Endpoint | Method | Description |
| ---- | ---- | ---- |
| `/` | GET | viewing interface |
| `/api/stats` | GET | stats |
| `/api/list?limit=&offset=` | GET | recent records |
| `/api/query?q=&k=` | GET | hybrid retrieval |
| `/api/put` | POST | `{title, body, tags, source?, weight?}` |
| `/api/delete` | POST | `{id}` |
| `/api/tidy` | POST | tidy up |
| `/api/ingest` | POST | `{path, split?}` |

## Architecture

```
memory_vault/
├── __main__.py / cli.py   命令行入口（12 个子命令）
├── config.py              配置加载（默认值 <- 文件 <- 环境变量 <- 参数）
├── vault.py               门面：协调存储/嵌入/索引/去重/压缩（进程内锁保证多线程一致）
├── store.py               SQLite 持久层：记录 CRUD、倒排词表、BM25 打分
├── vectors.py             向量索引：USearch -> numpy -> 纯 Python 三级降级
├── embedders.py           嵌入器工厂：local / sentence / api
├── ranking.py             融合排序：双通道 min-max 归一化 + 时间衰减
├── curation.py            去重决策（merge/skip）与摘要构建
├── markdown_io.py         markdown 导入导出（frontmatter / 拆分）
├── webapp.py              内置 Web 界面（http.server + 单页前端）
└── mcp_server.py          MCP stdio 服务（newline-delimited JSON-RPC）
```

Write flow: `put` → exact dedup by body (checksum) → semantic approximate dedup (top-k vector search) → persist + update index. Retrieval flow: `ask` → BM25 keyword score + semantic score → min-max normalized weighted fusion → time decay → rank and output.

## Development and Testing

```bash
python -m unittest discover -s tests -v   # 129 项测试：存储/检索/去重/压缩/markdown/CLI/MCP/Web/配置
pip install -e .                          # 可选：安装为命令 `vault`
```

Optional dependencies: `pip install usearch` (vector acceleration), `pip install sentence-transformers` (local semantic embeddings).

## License

[MIT](LICENSE)