xmemory
by Diadems666
README.md
# XMEMORY
[](https://github.com/Diadems666/xmemory/actions/workflows/ci.yml)
[](LICENSE)
A locally hosted, unified cross-referencing memory store bridging multiple AI
environments (Claude Desktop, Cursor, Kiro, Codex CLI, OpenCode, Gemini) with
a dark-mode dashboard for human management. See `SSOT.md` for the full
architecture spec and design notes (retention, hybrid search, injection
sanitization, scoping, write-time dedup, token budgets, contradiction
detection — all implemented).
## What's here
| File | Purpose |
| --- | --- |
| `xmemory_db.py` | Shared data layer: schema, embeddings, CRUD, hybrid (vector+FTS5) search, retention, backup/export/import, config |
| `init_db.py` | Creates/upgrades `xmemory.db` (WAL mode, `core_memory` + `vec_memory` + `fts_memory`) |
| `server.py` | MCP server (stdio) exposing 8 tools (see "MCP tools reference" below); sanitizes retrieved content |
| `dashboard.py` | FastAPI backend serving the dashboard UI + `/api/*` routes |
| `templates/index.html`, `static/` | Dark-mode dashboard UI (Memory Manager, Add Memory, Search Tester, Conflicts, Backup, Settings) |
| `config.json` | Runtime settings: `k_limit`, `token_budget_per_search`, `log_level`, `embedding_model` |
| `backups/` | Generated (gitignored) - timestamped `xmemory.db`/`config.json` snapshots from the Backup tab or `create_backup` |
| `startup.bat` | Launches the MCP server and the dashboard together |
| `dashboard_control.bat` | Double-click CLI menu to start/stop/restart/monitor just the dashboard |
| `dashboard_ctl.ps1` | PowerShell backend for `dashboard_control.bat` (process tracking, health check, log tailing) |
| `requirements.txt` | Top-level dependencies (`requirements-lock.txt` is the full frozen set that was verified to work) |
| `import_sources.py` | Re-runnable importer: pulls memories from Claude Code CLI and Codex CLI into XMEMORY (read-only on its sources) |
| `retention.py` | Manual/scheduled maintenance: archives stale, unreused `memory`-domain rows (dry-run by default, nothing auto-deleted) |
| `docs/xmemory_agent_rules.md` | Source of truth for agent behavior instructions - see "Agent instruction layer" below |
| `AGENTS.md`, `CLAUDE.md` + `.claude/xmemory_rules.md`, `.opencode/skills/xmemory.md`, `docs/chatgpt-desktop-instructions.md` | Per-client copies of those instructions, auto-discovered by each tool's own convention |
| `tests/smoke_test.py` | End-to-end test suite (store/search/dedup/scoping/conflicts/backup/export/import), self-cleaning, run by CI on every push and locally with `venv\Scripts\python tests\smoke_test.py` |
## Installation and setup
### Prerequisites
- **Windows 11** (this build's target; see `SSOT.md`). The Python code
itself is cross-platform, but `dashboard_control.bat`/`dashboard_ctl.ps1`
and `startup.bat` are Windows-specific launchers.
- **Python 3.11+** on PATH (`python --version`). Tested against 3.11.
- ~1 GB free disk for the venv + the `nomic-ai/nomic-embed-text-v1.5`
embedding model (~550MB, downloaded once, cached locally, then used
offline).
### Step by step
1. **Get the code.**
```bat
git clone https://github.com/<your-username>/xmemory.git C:\GIT\XMEMORY
cd C:\GIT\XMEMORY
```
(Or just use whatever local copy you already have at `C:\GIT\XMEMORY`
— that path is baked into `dashboard_control.bat`/`dashboard_ctl.ps1`'s
comments and this README's examples, but every script actually resolves
its own location at runtime, so a different path works fine too.)
2. **Create the virtual environment and install dependencies.**
```bat
python -m venv venv
venv\Scripts\pip install --upgrade pip
venv\Scripts\pip install -r requirements.txt
```
This installs `mcp`, `sqlite-vec`, `sqlcipher3-wheels` (SQLCipher, for
database encryption at rest — ships a prebuilt Windows wheel, no
compiler needed), `keyring` (OS-native key storage), `sentence-transformers`
(pulls in `torch`, the largest download here), `fastapi`, `uvicorn`,
`jinja2`, and `python-multipart`. `requirements-lock.txt` is the exact
frozen set verified to work together, if you hit a version conflict
with plain `requirements.txt`.
3. **Initialize the database.**
```bat
venv\Scripts\python init_db.py
```
Creates `xmemory.db` in WAL mode with `core_memory` (metadata),
`vec_memory` (`sqlite-vec`, 768-dim embeddings), and `fts_memory`
(FTS5 keyword index). Safe to re-run — it only creates what's missing
and migrates an older schema forward (see `xmemory_db.init_schema()`).
The file is SQLCipher-encrypted automatically from creation — a random
key is generated and stored via the OS keyring on first run, with no
extra step. See "Data protection: encryption and secret scanning" below
for what this covers and how to back up the key. (If you have an
*existing* pre-encryption `xmemory.db` from an older XMEMORY install,
run `venv\Scripts\python encrypt_db.py` once instead of `init_db.py` —
it migrates it in place, safety-backing up the plaintext original first.)
4. **Verify it actually works** before wiring anything up:
```bat
venv\Scripts\python -c "import xmemory_db as db; mid = db.store_memory('fact', 'setup-check', 'XMEMORY installed correctly.', 'setup'); print(db.hybrid_search('installed correctly')); db.delete_memory(mid)"
```
You should see a short list of search results printed (the memory you
just stored, ranked by relevance — you'll see others too if you've
already imported data), then no errors. This exercises the full path:
`sqlite-vec` extension loading, embedding model download-or-load, FTS5
indexing, hybrid search, and delete. The first run downloads the
embedding model (a minute or two depending on your connection); every
run after that is fully offline.
5. **Start it.** See "Running everything" below for `startup.bat` (MCP
server + dashboard together) or `dashboard_control.bat` (dashboard-only
control panel). Either way, the dashboard ends up at
**http://127.0.0.1:8765**.
6. **(Optional) Bring in memories that already exist elsewhere on this
machine.** See "Importing from other AI tools" below for the two
built-in sources (Claude Code CLI, Codex CLI), or hand `IMPORT_ENV_AI.md`
to an AI agent to survey the machine more broadly and import what it
finds, with your confirmation at each step.
7. **(Optional) Wire XMEMORY into your AI tools.** See "Integration"
below for exact config snippets per tool (Claude Desktop, Cursor, Kiro,
Claude Code CLI, Codex CLI, ChatGPT Desktop, OpenCode, and generic
stdio-MCP agents).
### Troubleshooting
- **`ModuleNotFoundError` for `mcp.server.fastmcp`** — you have `mcp` 2.x
installed (`FastMCP` was renamed to `MCPServer`); `server.py` already
uses the new import, so if you see this from your *own* code, update it
the same way.
- **`TypeError: unhashable type: 'dict'` from the dashboard** — a
Starlette/Jinja2 version mismatch in `TemplateResponse` call order;
`dashboard.py` already uses the current `(request, name, context)` form.
- **Port 8765 already in use** — `dashboard_ctl.ps1 -Action start` refuses
to start if something else is already listening there; check
`Get-CimInstance Win32_Process -Filter "Name='python.exe'"` for a stray
process, or change the port in `dashboard_control.bat`/`dashboard_ctl.ps1`
and `startup.bat` together (all three currently hardcode 8765).
- **`sqlite3.OperationalError: no such module: vec0`** — `sqlite_vec.load()`
didn't run, usually because you're using a `python.exe` outside the
venv (system Python may not allow extension loading, or won't have
`sqlite-vec` installed at all). Always use `venv\Scripts\python.exe`.
## Running everything
```bat
startup.bat
```
This launches:
1. `server.py` in its own console window (stdio MCP server — see note below).
2. The dashboard at **http://127.0.0.1:8765**.
You can also run each piece manually:
```bat
venv\Scripts\python server.py
venv\Scripts\python -m uvicorn dashboard:app --host 127.0.0.1 --port 8765
```
> **Note on the MCP server window:** `server.py` speaks MCP over stdio. In
> real usage, each client (Claude Desktop, Cursor, Kiro, ...) spawns its own
> `python server.py` process per the config below and talks to it directly
> over stdio — they do not connect to the window `startup.bat` opens. That
> window is included only so `startup.bat` satisfies "launch both at once"
> for manual/standalone testing; it's safe to close if you're only using the
> dashboard, and safe to leave running otherwise (idle, waiting on stdin).
## Dashboard control panel
Double-click **`dashboard_control.bat`** for an interactive menu to start,
stop, restart, and monitor the dashboard without touching the MCP server:
```
[1] Start dashboard [5] View recent logs
[2] Stop dashboard [6] Live tail logs (press any key to return)
[3] Restart dashboard [7] Open dashboard in browser
[4] Refresh status [0] Exit
```
It runs the dashboard as a hidden background process, tracks it via
`dashboard.pid`, and logs to `dashboard.out.log` / `dashboard.err.log`.
Closing the menu does **not** stop the dashboard — use option `[2]` first if
you want it down. It refuses to start if port 8765 is already held by an
unrelated process, and it won't double-start if it's already running.
## Importing from other AI tools
`import_sources.py` pulls existing memories from two sources already found
on this machine:
- **Claude Code CLI** — `~/.claude/projects/*/memory/*.md` (its own
per-project memory files).
- **Codex CLI** — `~/.codex/memories_1.sqlite` (`stage1_outputs` table,
Codex's own curated session-memory pipeline), opened via a read-only URI
connection so the live file is never locked or written.
Both sources are only ever **read** — nothing in `~/.claude` or `~/.codex`
is modified. Claude Code's memory `type` field (user/feedback/project/
reference) is normalized into XMEMORY's domain vocabulary so the
dashboard's domain filter stays meaningful — `user`/`reference` → `fact`,
`feedback` → `rule`, `project` → `memory`; the original type is preserved
as a `type:<original>` tag either way. Codex's session recaps import as
`memory`. See the module docstring in `import_sources.py` for the exact
mapping and field provenance.
The script is safe to re-run: every imported row carries a
`src:<tool>:<id>` tag, checked before insert, so running it again after new
Claude Code / Codex memories accumulate only imports what's new.
```bat
venv\Scripts\python import_sources.py --dry-run # preview counts, no writes
venv\Scripts\python import_sources.py # actually import
```
As of the last run: **75** Claude Code memories + **41** Codex memories =
**116** rows imported, breaking down as:
| domain | count | | source_agent | count |
| --- | --- | --- | --- | --- |
| `memory` | 97 | | `claude-code` | 75 |
| `rule` | 10 | | `codex-cli` | 41 |
| `fact` | 9 | | | |
## Hybrid search: vector + keyword fusion
`hybrid_search()` runs two rankings over the whole corpus and fuses them
with Reciprocal Rank Fusion (RRF) before applying the `k` cutoff:
- **Vector KNN** (`vec_memory`, cosine similarity) — catches semantic
matches even when the wording differs from what's stored.
- **FTS5 keyword search** (`fts_memory`, SQLite's built-in full-text index)
— catches exact-term matches (hostnames, error strings, package names)
that pure embedding similarity is often weak on.
Each result carries both `similarity` (cosine) and `fused_score` (the RRF
total); the dashboard's Search Tester shows whichever is more informative
per result. Domain/tag filters and the `archived` flag are applied as a
candidate set intersected with the fused ranking, not a pre-filter on one
side only — see `xmemory_db.hybrid_search()`'s docstring for the exact
mechanics.
## Embedding model
The default is `nomic-ai/nomic-embed-text-v1.5` (Apache 2.0, 768-dim,
`trust_remote_code=True`), loaded via `sentence-transformers` and cached
locally after the first run (~523MB on disk). It replaced the original
default, `all-MiniLM-L6-v2` (384-dim), because MiniLM hard-truncates input
at 256 tokens (~1000 characters) with no warning — a check against a
106-row sample of real stored memories found ~90% exceeded that limit
(median content length ~3112 characters), meaning most memories were being
silently embedded on a truncated prefix only. nomic-embed-text-v1.5
supports up to 8192 tokens, comfortably covering realistic memory content.
It uses asymmetric task prefixes per its training convention —
`search_document: ` is prepended to content being stored/compared,
`search_query: ` to search strings — handled automatically by
`xmemory_db.embed_text()`'s `task` parameter; nothing an operator needs to
manage. It requires `transformers<5.17` (a later release removed an
attention-mask helper the model's `trust_remote_code` modeling code still
calls — pinned in `requirements.txt`) and `einops`.
Switching `embedding_model` in `config.json` to a different model requires
re-embedding every stored memory, since old vectors from one model aren't
comparable to a different model's vector space, and `vec_memory`'s column
is fixed-dimension (a dimension change can't be applied in place). Run:
```
venv\Scripts\python reembed.py
```
after changing the config value. It takes its own safety backup first,
computes every new vector before touching the database, then rebuilds
`vec_memory` from scratch. Pass `--yes` to skip the confirmation prompt.
## Retrieved-content safety (MCP tool only)
`server.py`'s `hybrid_search` MCP tool treats retrieved memory content as
untrusted before handing it back to a consuming agent: known
prompt-injection patterns (`<|...|>` role markers, `[INST]`/`[SYSTEM]`
tags, `<system>`/`<assistant>`/`<user>` pseudo-tags, line-start
`SYSTEM:`/`ASSISTANT:`/`USER:` prefixes) are neutralized, each result is
capped at 1500 characters, and every result's content is prefixed with
`[xmemory#<id>]` so a consuming agent can tell "retrieved memory" apart
from live instructions. This only applies to the MCP tool — the dashboard's
`/api/search` is human-facing and shows content unmodified.
## Data protection: encryption and secret scanning
Two independent layers: one keeps secrets out of the database in the first
place, the other protects whatever content *is* stored.
**Encryption at rest.** `xmemory.db` is encrypted with SQLCipher (AES-256,
whole-database, applied below SQLite's query engine) — `hybrid_search`,
FTS5 keyword search, and vector search all work exactly as before, since
SQLCipher decrypts pages transparently before SQLite ever sees them. A
copied `xmemory.db` or `backups/*.db` file is unreadable without the
matching key — that's the point, but it also means:
- **The key is everything.** It's generated once and stored via `keyring`
(Windows Credential Manager/DPAPI on this platform, never a plaintext
file). Back it up before you need it: `venv\Scripts\python manage_key.py show`
prints it for you to save somewhere safe (a password manager, a printed
recovery sheet). `manage_key.py import` restores a previously-exported
key on a different machine/account, or after this one's keyring is lost.
**There is no other recovery path** — losing the key means losing every
backup made with it, permanently. This is a local-only tool with no
server-side escrow.
- **Headless/CI/containers** have no OS keyring session. Set the
`XMEMORY_DB_KEY` environment variable (64 hex characters) and
`xmemory_crypto.py` uses it instead of touching keyring at all — checked
first, every time, so it also works as a deliberate override on a normal
desktop (e.g. injecting the key from an external secrets manager).
- **Backups from before this feature** (an older XMEMORY install) are
plaintext; the dashboard's Backup tab and `list_backups()` still show
them, but `restore_backup()` refuses to restore one over the live
(encrypted) database, since that would silently downgrade it back to
plaintext.
**Write-time secret detection.** `store_memory`/`update_memory` scan new
content for known credential shapes (AWS/GitHub/Slack/Stripe/Google keys,
PEM private key blocks, JWTs, an assigned `api_key`/`password`/`token`)
before storing it, and reject the write if one matches — every client (MCP
tools, the dashboard, bulk import) goes through the same two functions, so
this is enforced uniformly, not just an instruction an agent might forget.
Like write-time deduplication, it never silently blocks or silently
strips content: it raises an error with a redacted preview so you can
judge it, and `force=True` (the same flag that already overrides a
duplicate warning) stores it anyway for a deliberate false positive (e.g.
a revoked example key in documentation). A secondary check for generic
high-entropy tokens exists but is off by default (Settings tab, or
`config.json`'s `secret_scanning.entropy_check_enabled`) — it catches
unrecognized random-looking secrets at the cost of flagging a lot of
ordinary technical content (filenames, code identifiers); the pattern-based
checks above stay active either way and had zero false positives against
this project's own real memory corpus.
## Retention: archiving stale memories
Finds `memory`-domain rows that are both old (default: 90+ days since
`updated_at`) and rarely/never reused (default: matched by `hybrid_search`
fewer than 1 time, tracked via `match_count`/`last_matched_at`), and
archives them — excluded from search, **not deleted**. Two equivalent
ways to run it, sharing the same `xmemory_db.py` implementation
(`find_retention_candidates`/`archive_many`/`purge_archived`):
- **CLI** (`retention.py`) — dry-run by default, nothing changes until you
pass `--apply`:
```bat
venv\Scripts\python retention.py # preview candidates
venv\Scripts\python retention.py --apply # archive them
venv\Scripts\python retention.py --archive-after-days 30 --min-match-count-to-keep 2
venv\Scripts\python retention.py --purge # preview already-archived rows
venv\Scripts\python retention.py --purge --apply # permanently delete them
```
- **Dashboard** (Backup tab's Retention panel) — set the same two
thresholds, "Find Candidates", review the list (each row expandable,
all pre-checked), deselect any you want to keep, "Archive Selected"
(confirms first). No dashboard UI for `--purge` yet — permanent deletion
of already-archived rows is CLI-only for now, a deliberate extra step
of friction for an irreversible action.
Archived rows can also be managed from the dashboard's Memory Manager tab
(Archive/Restore buttons per row, "Show archived" toggle) regardless of
which path archived them. `rule`/`fact` rows are never touched by
retention — only episodic `memory`-domain rows decay this way, since
standing rules and facts don't go stale on a schedule.
## Global / local / project-aware scoping
`core_memory.scope` is `'global'` (applies everywhere) or a project slug.
It's not a hard filter — `hybrid_search`'s `project` parameter (MCP tool)
or the Search Tester's "Project scope" field gives same-project and global
memories a small ranking boost, so they surface ahead of other-project
noise without hiding a genuinely strong match from elsewhere. Set it per
memory via the Add Memory form's Scope field (defaults to `global`) or the
Memory Manager's Edit action; filter the grid by scope via the toolbar
dropdown.
## Write-time duplicate detection
`store_memory`/`update_memory` check new content against existing active
same-domain memories before writing; a ≥0.90 cosine-similarity match is
rejected with a pointer to the existing memory's id instead of being
stored, preventing the corpus from filling up with near-identical restated
facts over time. Override with `force=true` (MCP tools) or the dashboard's
confirm-to-override prompt when you genuinely want both.
## Possible duplicates / conflicts
The dashboard's **Conflicts** tab surfaces same-domain, same-scope memory
pairs whose content is highly similar (cosine ≥ 0.90) but weren't caught
at write time (e.g. two separately-imported memories, or two that drifted
apart in scope before growing similar again) — detection only, nothing
resolves automatically. Pick which side of a pair stays current; the other
gets archived and the kept memory's `supersedes` field records the
replacement, so nothing is silently lost. Not exposed as an MCP tool on
purpose — resolving a contradiction is a human call.
## Token-budget-aware search results (MCP tool only)
On top of the per-result 1500-character cap and the `k` result-count cap,
`hybrid_search`'s MCP tool now also caps the *total* approximate token
cost across all returned results combined (`config.json`'s
`token_budget_per_search`, default ~2000, editable from the dashboard's
Settings tab). Results are added in ranked order until the next one would
exceed budget, then stop — except the single best result is always
included even if it alone exceeds budget, so a real match is never
withheld outright.
## Backup, export, and import
Two different mechanisms, for two different needs:
- **Backup/restore** (dashboard's **Backup** tab, or the `create_backup`
MCP tool) — a full raw snapshot of `xmemory.db` (WAL-checkpointed first)
+ `config.json`, byte-for-byte, including vectors and the FTS5 index.
This is disaster recovery: "put everything back exactly as it was."
Timestamped into `backups/`. Creating a backup is non-destructive (only
ever adds a file) and available as an MCP tool. **Restoring** overwrites
the live database — dashboard-only, never an MCP tool — and always takes
its own safety backup of current state first, so a mistaken restore is
itself undoable. The dashboard also lets you download a backup file
directly.
- **Export/import** (dashboard's **Backup** tab, or the `export_memories`/
`import_memories` MCP tools) — a portable JSON format of memory content
(domain, tags, content, source_agent, scope, updated_at — **no
vectors**, those get regenerated on import via the normal embedding
path). This is for moving or merging a *subset* of memories: between
machines, between embedding models, or just to inspect/share what's
stored. Export takes the same `domain`/`tags`/`scope`/`include_archived`
filters as search. Import runs every record through the normal
`store_memory` path, so the same ≥0.90 dedup check applies — duplicates
are skipped and reported, not silently re-added (pass `force` to
bypass). Import is purely additive; it never deletes or overwrites
existing memories.
```bat
:: backup (dashboard-equivalent, or use the Backup tab)
venv\Scripts\python -c "import xmemory_db as db; print(db.create_backup())"
```
The dashboard's Backup tab covers all of this without touching a terminal:
create/list/download/restore backups, and export-with-filters /
import-from-file forms with a result summary (imported / skipped
duplicates / failed, with reasons).
## Verified behavior
The dashboard and DB layer were smoke-tested end-to-end during
implementation: store → hybrid search (vector+FTS5 fusion, with similarity
and fused scoring) → update (re-embeds and re-indexes automatically) →
delete (removes the `core_memory`, `vec_memory`, and `fts_memory` rows) →
archive/restore → backup/restore (including the Windows WAL-file-lock
retry path) → export/import (including the dedup interaction) → settings
read/write. All confirmed working against this repo's `venv`, both via
direct Python calls and through the actual dashboard UI in a real browser
(including the file-upload import flow, using a synthetic `File`/
`DataTransfer` object). `import_sources.py` was run in `--dry-run` mode first to
verify counts/mapping, then for real, then re-run to confirm it's
idempotent (0 new rows on the second pass). The schema migration (adding
`archived`/`match_count`/`last_matched_at`/`scope`/`supersedes` columns and
the `fts_memory` table, then backfilling FTS5 and scope for the existing
116 rows) was verified against the live database, not just a fresh one.
`retention.py` was run in dry-run mode against the real dataset. The MCP
server's injection sanitizer was verified against a crafted
`<|im_start|>system...<|im_end|>` payload. Write-time dedup was verified
for identical content (rejected), genuinely different content (accepted),
and the `force` override (bypasses correctly). Conflict detection/
resolution was verified against both a crafted near-duplicate pair and the
real corpus. Scoping was verified live: a project-scoped search correctly
re-ranked matching-scope results ahead of others. All of the above were
also exercised through the actual dashboard UI (not just the Python API)
in a real browser, including catching and fixing a UI bug along the way
(the Add Memory scope field's pre-filled default value would silently
concatenate with typed input instead of being replaced — fixed to use a
placeholder instead).
`tests/smoke_test.py` now codifies the core of this into a repeatable,
self-cleaning suite (store/search/delete, dedup rejection + force
override, scope-bonus ranking, conflict detection + resolution, backup +
restore round-trip, export + import round-trip) that CI runs on every
push. Building it surfaced one genuine, worth-knowing scoping nuance: the
`project` ranking bonus applies to *every* `global`-scoped row, not just
relevant ones — so in a populated corpus, a row scoped to a specific
*different* project can legitimately rank below unrelated `global` content
in a small top-k window, even when it's semantically closer to the query.
That's consistent with the intended design (prefer your own project +
global defaults over a different specific project), not a bug, but it's
worth knowing if a search from inside a project feels like it's
surfacing "unrelated global stuff" ahead of a highly specific match from
another project.
## Agent instruction layer
Wiring the MCP server into a client (see "Integration" below) makes the
*tools* available, but doesn't tell an agent *when or how* to use them well
— that's a separate, smaller problem worth solving explicitly, since a
tool an agent doesn't reliably reach for is close to useless.
**`docs/xmemory_agent_rules.md` is the single source of truth** for that
behavior guidance. Everything else is a generated copy, placed wherever
each client auto-discovers instructions by its own convention, so nothing
extra needs configuring beyond the MCP wiring itself:
| File | Auto-discovered by |
| --- | --- |
| `AGENTS.md` (repo root) | Codex CLI, and the general `AGENTS.md` convention several other tools follow |
| `CLAUDE.md` (repo root) | Claude Code CLI — a 3-line pointer using Claude Code's `@path` file-import syntax to pull in `.claude/xmemory_rules.md` |
| `.claude/xmemory_rules.md` | The file `CLAUDE.md` imports — same content as `AGENTS.md`, kept as its own file so the import target is unambiguous |
| `.opencode/skills/xmemory.md` | OpenCode's skills mechanism (has YAML frontmatter `name`/`description` OpenCode uses to decide when to surface it) |
| `docs/chatgpt-desktop-instructions.md` | Nothing automatic — ChatGPT doesn't have a file-based auto-discovery convention; paste this into Custom Instructions manually. See "ChatGPT Desktop" under Integration for the separate (and non-obvious) question of how the *connection* itself works. |
The core instructions (a "Universal Artifact" — Search-before-acting,
project-scoped search, dedup/duplicate handling, the real `domain`
vocabulary, and memory-poisoning/conflict-resolution security guardrails)
are **kept byte-identical** across `AGENTS.md`, `.claude/xmemory_rules.md`,
and `.opencode/skills/xmemory.md` on purpose: an exact-match static prefix
is what makes prompt caching effective, so this text shouldn't be reworded
or reordered per-client. If you edit the behavior guidance, edit
`docs/xmemory_agent_rules.md` first, then propagate the same change
identically into the three artifact copies and into
`docs/chatgpt-desktop-instructions.md`'s paraphrased version.
### Active learning (opt-in, off by default)
`config.json`'s `active_learning` block is a feature flag, not new code —
when `enabled: true`, agents following the instruction layer above inject
an extra "ask before storing if uncertain" protocol into their own
behavior (human-in-the-loop confirmation for borderline memories). No new
MCP tool: the gating is instruction-driven client-side, specifically to
avoid growing the tool surface (more tools = more prompt-prefix cost = a
worse-caching prefix). See `docs/xmemory_agent_rules.md`'s "Active
Learning Protocol" section for the exact injected text and rollout
guidance, and "Configuration" below for the schema.
## MCP tools reference
`server.py` exposes 8 tools over stdio. All write tools return a compact
JSON string (never raise on expected conditions like a duplicate or a
missing id - errors come back as `{"status":"error"/"duplicate", ...}` so
an agent can branch on them without a try/catch). Three tools deliberately
have **no** MCP equivalent — see "Why some operations are dashboard-only"
below.
| Tool | Parameters | Returns | Notes |
| --- | --- | --- | --- |
| `store_memory` | `domain, tags, content, source_agent, force=False` | `{"id", "status":"stored"}`, `{"status":"duplicate", "existing_id", "similarity", "detail"}`, or `{"status":"secret_detected", "label", "redacted", "detail"}` | Embeds `content`, indexes it into FTS5, defaults `scope` to `'global'` (use `update_memory` after to set a project scope). Rejects ≥0.90 cosine-similar existing content, and content matching a known credential shape (see "Data protection" above), unless `force=True` (overrides both). |
| `hybrid_search` | `query, domain="", tags="", project=""` | JSON array of `{"id","domain","tags","content","source_agent","scope","score"}` | Vector+FTS5 RRF fusion. `project` nudges same-project/global results ahead, doesn't filter. Capped by `k_limit` (config, default 5) **and** `token_budget_per_search` (config, default ~2000 approx tokens) — may return fewer than k results if budget runs out, but always at least 1 if there's a match. Each result's `content` is injection-sanitized, 1500-char capped, and prefixed `[xmemory#<id>]`. Matched rows' `match_count`/`last_matched_at` are bumped (used by `retention.py`). |
| `update_memory` | `id, domain="", tags="", content="", source_agent="", scope="", supersedes=0, force=False` | `{"id", "status":"updated"}` or a `"duplicate"`/`"secret_detected"`/`"error"` status | Only non-empty/non-zero fields change. Re-embeds and re-indexes FTS5 only if `content` changes (and only then runs the dedup and secret-detection checks). `supersedes` marks this memory as replacing another by id — does not archive the replaced one itself (use the dashboard's Conflicts tab for that combined action, or `archive_memory` isn't exposed to MCP - do it from the dashboard). |
| `delete_memory` | `id` | `{"id", "status":"deleted"}` | Removes the `core_memory`, `vec_memory`, and `fts_memory` rows. Not soft — there's no MCP-exposed archive/restore; use `delete_memory` sparingly and lean on the dashboard's Archive button for anything you might want back. |
| `list_memories` | *(none)* | JSON array of all active memories (metadata + content, no vectors) | No pagination/limit - fine at hundreds of rows, will need one if the corpus grows to many thousands. |
| `export_memories` | `domain="", tags="", scope="", include_archived=False` | JSON array (no vectors) | Read-only. Same filters as search; use to hand a subset of XMEMORY to another tool/process or inspect current contents. |
| `import_memories` | `data` (JSON string, array of objects with at least `content`), `force=False` | `{"imported", "skipped_duplicates", "failed"}` | Purely additive - never overwrites/deletes. Each record goes through `store_memory`'s normal embed+dedup+secret-detection+index path. A duplicate is silently skipped and counted; a detected secret lands in `failed` with the reason. `force=True` bypasses both checks for the whole batch. |
| `create_backup` | *(none)* | `{"backup_file", "config_file", "size_bytes", "row_count", "created_at"}` | Non-destructive snapshot of `xmemory.db`+`config.json` into `backups/`. Good practice to call before a large `import_memories` batch. |
### Why some operations are dashboard-only
Three operations exist in `xmemory_db.py` and the dashboard, but are
**not** MCP tools, on purpose — each overwrites or resolves something in a
way that should be a human decision, not something an unattended agent
triggers:
- **`restore_backup`** — overwrites the entire live database. An agent
accidentally (or via a confused/adversarial prompt) restoring a stale
backup would silently roll back everything written since. Dashboard's
Backup tab only, and it double-confirms (browser `confirm()`, plus its
own automatic safety-backup-before-restoring).
- **`resolve_conflict`** — decides which of two contradictory memories is
correct and archives the other. Detecting a possible contradiction is
cheap and safe to automate (that's what `find_possible_conflicts` /
the Conflicts tab do); *resolving* one requires judgment about which
claim is actually true, which is exactly the kind of thing this project
doesn't want an agent silently doing to its own knowledge base.
- **`archive_memory`/`restore_memory`** (per-row, not the DB-wide
restore above) — not currently exposed either; still small/reversible
enough that this is more a consistency choice (keep row-level
curation actions dashboard-only alongside the two above) than a hard
safety requirement. Worth revisiting if an agent-driven retention
workflow becomes useful later.
## Integration
> **Recommended for every client below:** set `PYTHONUNBUFFERED=1` in the
> server's environment. Python buffers stdout by default when it's not
> attached to a terminal (i.e. always, for a stdio MCP subprocess), which
> can delay JSON-RPC responses reaching the client; combined with
> `server.py`'s own UTF-8 stdio hardening (Windows defaults stdout/stderr
> to the locale codepage, e.g. cp1252, which can corrupt UTF-8 payloads),
> this keeps the stdio transport reliable on Windows. The JSON snippets
> below include it; add the equivalent for any client not shown.
### Claude Desktop
Edit `%APPDATA%\Claude\claude_desktop_config.json` and add an entry under
`mcpServers`:
```json
{
"mcpServers": {
"xmemory": {
"command": "C:\\GIT\\XMEMORY\\venv\\Scripts\\python.exe",
"args": ["C:\\GIT\\XMEMORY\\server.py"],
"env": { "PYTHONUNBUFFERED": "1" }
}
}
}
```
Restart Claude Desktop after saving. All 8 tools from the "MCP tools
reference" above (`store_memory`, `hybrid_search`, `update_memory`,
`delete_memory`, `list_memories`, `export_memories`, `import_memories`,
`create_backup`) will appear as available MCP tools.
### Claude Code CLI
Distinct from Claude Desktop above — this is the CLI tool. Easiest is the
official command (run from anywhere, registers it at user scope so it's
available in every project):
```bat
claude mcp add --scope user -e PYTHONUNBUFFERED=1 xmemory -- C:\GIT\XMEMORY\venv\Scripts\python.exe C:\GIT\XMEMORY\server.py
```
Or edit the config JSON directly — Claude Code CLI reads MCP servers from
`~/.claude/mcp.json` (global) or a project-local `.mcp.json` in a repo
root (same `mcpServers` schema as Claude Desktop above):
```json
{
"mcpServers": {
"xmemory": {
"command": "C:\\GIT\\XMEMORY\\venv\\Scripts\\python.exe",
"args": ["C:\\GIT\\XMEMORY\\server.py"],
"env": { "PYTHONUNBUFFERED": "1" }
}
}
}
```
Run `claude mcp list` to confirm it's registered, then restart/reload any
running `claude` session to pick it up.
### Cursor
Cursor reads MCP servers from `.cursor/mcp.json` (project-level) or the
global `~/.cursor/mcp.json`. Add:
```json
{
"mcpServers": {
"xmemory": {
"command": "C:\\GIT\\XMEMORY\\venv\\Scripts\\python.exe",
"args": ["C:\\GIT\\XMEMORY\\server.py"],
"env": { "PYTHONUNBUFFERED": "1" }
}
}
}
```
### Kiro
In Kiro's MCP settings (Kiro > Settings > MCP Servers, or the workspace
`.kiro/settings/mcp.json`), add the same `command`/`args`/`env` as above
under a `xmemory` key.
### OpenCode — already wired up
`%APPDATA%\opencode\opencode.json` has an `xmemory` entry under `mcp`
(alongside the pre-existing `tolaria` and `open-knowledge` servers, both
left untouched):
```json
"xmemory": {
"type": "local",
"enabled": true,
"command": [
"C:\\GIT\\XMEMORY\\venv\\Scripts\\python.exe",
"C:\\GIT\\XMEMORY\\server.py"
],
"environment": { "PYTHONUNBUFFERED": "1" }
}
```
A backup of the pre-edit config was saved alongside it
(`opencode.json.bak-xmemory-<timestamp>`). Restart OpenCode to pick it up.
Re-verified 2026-09-17: the config still points at real, current files
(`venv\Scripts\python.exe` and `server.py` both exist), `server.py` starts
cleanly with no errors, and all 8 tools (including the newer
`export_memories`/`import_memories`/`create_backup`) are registered on it
— no drift since the original wiring. `PYTHONUNBUFFERED=1` was added to
the live config as part of this same pass (it had been recorded as done
in an earlier memory entry that turned out not to match reality — fixed
both the config and the memory).
### Codex CLI
Codex CLI uses TOML, not JSON, and its `command`/`args` split the
executable and its arguments differently — add a `[mcp_servers.xmemory]`
table to `~/.codex/config.toml`:
```toml
[mcp_servers.xmemory]
command = "C:\\GIT\\XMEMORY\\venv\\Scripts\\python.exe"
args = ["C:\\GIT\\XMEMORY\\server.py"]
[mcp_servers.xmemory.env]
PYTHONUNBUFFERED = "1"
```
Run `codex mcp list` to confirm it's registered, and restart any running
Codex session to pick it up.
### ChatGPT Desktop
Two different integration paths, and they're not interchangeable:
- **If you use ChatGPT Desktop in its Codex-connected/agent mode**, it
reads MCP servers from the **same** `~/.codex/config.toml` as Codex CLI
above — the `[mcp_servers.xmemory]` entry you just added already covers
it, nothing extra to configure.
- **If you use ChatGPT's web-style "Developer Mode" custom connectors**
(Settings → Apps & Connectors → Advanced Settings → Developer Mode →
Create), that path is **remote-HTTPS-and-OAuth only** — it does not
connect to a local stdio process the way Claude Desktop/Cursor/Codex CLI
do. `server.py` only implements stdio transport today, so it can't be
wired into a Developer Mode connector without additional work: exposing
it over `streamable-http` (the `mcp` package supports this transport;
`server.py` would need a code change to use it instead of `stdio`) and a
tunnel (e.g. Cloudflare Tunnel, ngrok, or OpenAI's own Secure MCP
Tunnel) to make that endpoint reachable from OpenAI's servers, plus
OAuth. That's real added infrastructure for a tool meant to stay local
and single-machine (see `SSOT.md`) — not done here, and not recommended
unless you specifically need ChatGPT's web/mobile clients (not the
desktop app) to reach XMEMORY too.
### Gemini CLI / other stdio-MCP agents
Any other agent that supports stdio MCP servers can attach the same way:
point its MCP config at `command: C:\GIT\XMEMORY\venv\Scripts\python.exe`
with `args: ["C:\GIT\XMEMORY\server.py"]` (or the TOML equivalent if it
follows Codex CLI's config style). For Dockerized agents, volume-map
`C:\GIT\XMEMORY` into the container so the agent can reach the shared
`xmemory.db` file and run `server.py` (with the container's own Python
environment, or by also mounting `venv`), keeping in mind SQLite's WAL mode
allows safe concurrent access from multiple processes/containers as long as
they all use the same `xmemory.db` path.
## Configuration
`config.json` controls:
- `k_limit` — hard cap on results returned by `hybrid_search` (default 5, prevents token bloat).
- `token_budget_per_search` — approximate total token cap (chars÷4) across all of `hybrid_search`'s MCP-tool results combined (default 2000). See "Token-budget-aware search results" above.
- `log_level` — reserved for future logging verbosity control.
- `embedding_model` — the `sentence-transformers` model name (default `nomic-ai/nomic-embed-text-v1.5`, 768 dims — see "Embedding model" above). Changing this requires re-embedding existing memories: run `venv\Scripts\python reembed.py` after updating the value, which rebuilds `vec_memory` for every row with the new model (takes its own safety backup first).
- `active_learning` — opt-in feature flag block for the instruction-driven "ask before storing an uncertain memory" behavior (see "Agent instruction layer" above). Default `{"enabled": false, "scope": "project", "ask_before_store": true, "max_questions_per_session": 3, "confidence_threshold": 0.8}`. `server.py` logs the effective values at startup; flipping `enabled` requires no code change, just editing this block (directly, or by asking an agent following `docs/xmemory_agent_rules.md` to do so) and restarting the MCP server process.
Edit `config.json` directly, or use the **Settings** tab in the dashboard
(`k_limit`, `token_budget_per_search`, and `log_level` — `active_learning`
isn't in the dashboard UI yet, edit the file for that one).
This server cannot be deployed
Maintenance
ActivityMaintained
ResponsivenessNo issues