# Contextual Memory MCP

A local-first persistent contextual-memory engine with an MCP adapter for AI
models such as Qwen.

The project is being developed incrementally. The first ingestion target is a
directory of reusable Markdown prompt material, but the storage and retrieval
layers are general-purpose.

## Architecture

```text
Text, images, audio, and video
          |
          v
Modality-specific semantic distillation
          |
          +--> SQLite semantic capsules and sparse concepts
          |
          +--> packed int8 latent parameter bank
          |
          v
Hybrid retrieval and context assembly
          |
          v
Thin MCP adapter used by Qwen
```

The CLI performs administrative work. The MCP server exposes memory operations
that are useful to the model.

## Install with uv

```bash
uv venv --python 3.12
uv pip install -e .
cp .env.example .env
```

Python 3.12 or 3.13 is recommended for broad compatibility with the current
machine-learning dependencies.

## Index files and directories

The CLI accepts either one file or an entire directory. With no `--name`, the
content is added to the normal writable `main` store:

```bash
uv run contextual-memory-index scan /path/to/character.md
uv run contextual-memory-index scan /path/to/saved/prompts
```

Provide `--name` to add the target to a named database. If that database does
not exist, it is created automatically:

```bash
uv run contextual-memory-index scan /path/to/character.md --name roleplay
```

New named databases are immutable after indexing by default. Use `--mutable`
when the named database should remain writable:

```bash
uv run contextual-memory-index scan /path/to/saved/prompts --name prompt-library --mutable
```

A later explicit scan can update an existing named database. Locked databases
are reopened only for that update and then returned to their previous mode.
Use `--replace` when the entire named database should be rebuilt instead:

```bash
uv run contextual-memory-index scan /path/to/saved/prompts --name prompt-library --replace
```

Directory scans support exclusions, and either kind of scan can force unchanged
content to be indexed again:

```bash
uv run contextual-memory-index scan /path/to/saved/prompts --exclude archive --force
```

## Clear stored data

Interactive confirmation:

```bash
uv run contextual-memory-index clear --store main
```

Non-interactive confirmation:

```bash
uv run contextual-memory-index clear --store main --yes
```

Scanning and clearing are deliberately CLI-only maintenance operations. They are
not exposed to the model through MCP.

## Run the MCP server

```bash
uv run contextual-memory-mcp
```

The MCP server exposes model-facing recall, ingestion, lifecycle, and maintenance tools:

### `recall_memory`

Retrieves semantically, lexically, and graph-related memory and returns clean,
source-attributed Markdown that Qwen can use in its active context.

Typical model call:

```text
recall_memory(
  query="Create a gender-swapped character reference sheet while preserving identity, colors, and species traits"
)
```

### `store_memory`

Automatically captures durable information directly stated by the user. The
model is instructed to call it before drafting its response whenever the current
message contains durable information, including during ordinary conversation. The user does not need to say
“remember this,” and routine storage must not trigger a permission question or
an announcement. The model supplies an integer memory type and a normalized
importance score from 0.0 to 2.0. The server validates those values and assigns
lifecycle state, origin, confidence, source quality, and conservative
server-owned lifecycle policy.

### `store_memory_candidate`

Stores model-generated hypotheses and interpretations as candidate inferences.
It is not a substitute for `store_memory` when the user directly stated the
information, whenever that statement is durable. The
model supplies semantic type and importance, while the server assigns lifecycle
state, origin, confidence, source quality, and conservative retention policy.

### `explore_memory`

Traverses the knowledge graph around a concept and returns related concepts plus
supporting memory excerpts.

The MCP does **not** expose scan, clear, stats, or database-deletion tools. Those
are administrative concerns and remain under direct user control.

## LM Studio configuration

Add the server as a stdio MCP using the repository's virtual environment. A
typical configuration resembles:

```json
{
  "mcpServers": {
    "contextual-memory": {
      "command": "/absolute/path/to/contextual-memory-mcp/.venv/bin/contextual-memory-mcp",
      "args": [],
      "cwd": "/absolute/path/to/contextual-memory-mcp"
    }
  }
}
```

Alternatively, invoke it through uv:

```json
{
  "mcpServers": {
    "contextual-memory": {
      "command": "uv",
      "args": [
        "run",
        "--directory",
        "/absolute/path/to/contextual-memory-mcp",
        "contextual-memory-mcp"
      ]
    }
  }
}
```

Use absolute paths when LM Studio is launched outside a terminal, because GUI
applications may not inherit the same shell PATH.

## Storage

Persistent data is stored beneath `CM_DATA_DIR`, which defaults to `./data`:

- `contextual_memory.sqlite3`
- `vectors/latent_vectors.sqlite3`(compact int8 parameter bank)
- `store_overlays.sqlite3` (local ranking overlays for immutable stores)
- `stores/<name>/manifest.json` plus each scanned store database

The embedding model is stored in the application's own model directory and normal
server/index runs load it with network access disabled. Install it once before the
first scan or recall:

```bash
uv run contextual-memory-index model install
```

Check the local installation without contacting Hugging Face:

```bash
uv run contextual-memory-index model status
```

The default location is `CM_DATA_DIR/models`. Override it with `CM_MODELS_DIR`.
After installation, deleting the global Hugging Face cache does not affect this
project. A missing model produces an explicit error instead of downloading during
an MCP or indexing operation.

## Current retrieval path

`recall_memory` currently combines:

- semantic similarity from the packed int8 parameter bank
- SQLite full-text relevance
- graph relationships
- segment importance
- per-source diversity
- a configurable context-size budget

The returned value is Markdown rather than an embedding blob or database
record, so the model receives reusable source material directly.


## Memory importance and lifecycle

Memory importance is persistent and evolves through an explicit maintenance pass:

```text
new recall accesses
      +
configured access gain
      -
elapsed inactivity decay
      |
      v
updated importance
      |
      v
promotion / archival policy
```

Run `run_memory_maintenance` through MCP to reinforce accessed memories, decay
inactive unpinned memories, synchronize vector metadata, and then evaluate
candidate promotion or active-memory archival. Pass `dry_run=true` to inspect
all proposed changes without mutating storage.

All state, type, origin, lifecycle-reason, and importance-reason enums are
stored as SQLite integers and exposed in Python as `IntEnum` values.


## Multiple memory stores

The server provides a writable `main` store. Scanned databases live beneath
`data/stores/<name>/` and contain their own `manifest.json`. The filesystem is
the source of truth: there is no separate store registry, registration step,
mount operation, or startup fixture.

`list_memory_stores` discovers manifests directly from disk. A named store is
opened lazily when recall or another operation addresses it. Returned memory
IDs are globally qualified, for example `ghidra-core:seg_abcd`.

Store modes are SQLite integers exposed through `StoreMode`:

```text
0 READ_WRITE
1 READ_ONLY
2 IMMUTABLE
```

Writes require an explicit `target_store`; they are never broadcast. Before a
write, clients must call `list_memory_stores` whenever the destination is not
already established in the current context. Store IDs must not be guessed or
discovered by intentionally causing a failed write.

Maintenance runs only against writable stores. Read-only and immutable SQLite
databases are opened with SQLite read-only URI modes and are never migrated
automatically. Their recall access counts and user-specific ranking choices are
kept in `store_overlays.sqlite3` instead of changing canonical content.

Creating a scanned database is sufficient to make it discoverable:

```bash
uv run contextual-memory-index scan ./project --name project-memory
```

This creates:

```text
data/stores/project-memory/
├── manifest.json
├── project-memory.sqlite3
└── vectors/
```

No registration or mounting command is required.


## Web search providers

Automatic web acquisition can use multiple search providers in priority order. Providers that need credentials are skipped unless configured, and failures automatically fall through to the next provider.

```env
CM_WEB_SEARCH_PROVIDERS=exa,brave,tavily,searxng,duckduckgo
CM_WEB_SEARCH_TIMEOUT=12
CM_EXA_API_KEY=
CM_BRAVE_SEARCH_API_KEY=
CM_TAVILY_API_KEY=
CM_SEARXNG_URL=http://localhost:8080
```

With no API keys or SearXNG URL, DuckDuckGo is used as the zero-configuration fallback. For a self-hosted setup, place `searxng` first and set `CM_SEARXNG_URL`. The configured SearXNG instance must permit JSON search responses.


## Web acquisition and indexing

By default, indexed memory is authoritative over the web. `recall_memory` only triggers web discovery, page ingestion, and a second local recall when local retrieval returns no hits. Search providers are attempted in configured priority order; unconfigured providers are skipped and failures fall through to the next provider.

```env
CM_MEMORY_AUTHORITY_OVER_WEB=true
CM_WEB_SEARCH_PROVIDERS=exa,brave,tavily,searxng,duckduckgo
CM_WEB_SEARCH_CACHE_DAYS=7
CM_WEB_ACQUISITION_RETRY_DAYS=7
CM_WEB_ACQUISITION_REFRESH_DAYS=90
```

Search results are cached in `data/web_acquisition.sqlite3`. Acquisition history suppresses repeated searches for unavailable topics and schedules successful topics for a later refresh.

The index CLI accepts directories, individual files, direct URLs, and search queries:

```bash
python main.py scan ./prompts
python main.py scan ./character.md --name roleplay
python main.py scan https://example.org/wiki/Character --name roleplay
python main.py scan --search "Character franchise personality dialogue" --name roleplay
```

When `--name` is omitted, content is indexed into the writable `main` store. Missing named stores are created automatically and are immutable after indexing unless `--mutable` is supplied. MediaWiki pages use their API when available, and GitHub `blob` URLs are fetched through their raw-content endpoint.

## Visual memory

`scan` now discovers common image formats and sends them to an OpenAI-compatible
vision model. LM Studio's local server is the default endpoint.

```bash
contextual-memory-index scan ./reference-images \
  --name visual-reference \
  --vision-model qwen2.5-vl-7b-instruct
```

The extractor stores a searchable visual summary plus independently classified
text regions. Its general-purpose taxonomy distinguishes speech and thought
bubbles, captions/overlays, clothing text, text attached to physical surfaces,
documents, UI/HUD text, sound effects, watermarks, and uncategorized text.
Every region includes normalized bounding coordinates and separate OCR and
classification confidence values. This lets retrieval distinguish, for example,
a caption saying "red coat" from the words printed on the coat itself.

Configuration:

- `CM_VISION_MODEL`: default vision model name (required when images are found)
- `CM_VISION_BASE_URL`: compatible `/v1` endpoint; defaults to LM Studio at
  `http://127.0.0.1:1234/v1`
- `CM_VISION_API_KEY`: optional bearer token
- `CM_VISION_TIMEOUT`: request timeout in seconds
- `CM_VISUAL_GLOBS`: comma-separated image globs

Use `--no-vision` to skip image files during a directory scan. Scanning a single
image with `--no-vision` is rejected rather than silently doing nothing.

## Latent multimodal memory

This branch stores learned external-memory representations rather than copies of
source training material. The scanner reads a source only during ingestion,
distills it into a lossy semantic capsule, embeds that capsule, quantizes the
normalized embedding to signed int8, and persists the compact parameter record.
Raw image, audio, and video bytes are never copied into the memory store.

Supported inputs:

- Text: works immediately through the built-in heuristic semantic provider.
  Configure `CM_SEMANTIC_MODEL` later for higher-quality local-LLM distillation.
- Images: analyzed only when `CM_VISION_MODEL` is configured, including classified
  text regions. Otherwise directory scans skip images and report the count.
- Audio: transcribed only when `CM_AUDIO_MODEL` is configured, then semantically
  distilled. Otherwise directory scans skip audio and report the count.
- Video: enabled only when both vision and audio providers are configured. It is
  sampled with ffmpeg, frames are analyzed, audio is transcribed, and both channels
  become a compact temporal capsule.

Optional local services are enabled per modality:

```dotenv
CM_SEMANTIC_PROVIDER=auto
# CM_SEMANTIC_MODEL=your-local-instruct-model
# CM_VISION_MODEL=your-local-vision-model
# CM_AUDIO_MODEL=your-local-whisper-model
```

Configured endpoints use the OpenAI-compatible API and default to LM Studio's
local `http://127.0.0.1:1234/v1`. Unconfigured modalities do not prevent the
server or text scanner from running. Audio/video ingestion requires `ffmpeg` on
`PATH` only when those modalities are actually processed. The original wording or media cannot be reconstructed from the stored
capsules and quantized vectors; retain source files separately when exact
reproduction or citation is required.

### Parameter storage

The former vector collection has been replaced by
`latent_vectors.sqlite3`. Each row contains only a stable memory ID, dimensions,
a scalar quantization scale, and a packed int8 vector. Retrieval uses exact
integer-vector scanning and SQLite remains authoritative for semantic capsules,
concept weights, and lifecycle state.

### Hierarchical latent distillation

Large text files are distilled hierarchically rather than sent to the semantic
model in one request. Source text is split into bounded input chunks, chunk
capsules are merged into section capsules, and section capsules are merged into
a document capsule. The stored latent document contains the compact document
capsule and section capsules; original source wording and transient leaf
capsules are not retained.

Configuration:

```dotenv
CM_SEMANTIC_MAX_TOKENS=512
CM_SEMANTIC_INPUT_CHUNK_CHARS=12000
CM_SEMANTIC_MERGE_FAN_IN=6
CM_SEMANTIC_CONCURRENCY=1  # 1 = sequential; 2+ enables bounded parallel requests
```

`CM_SEMANTIC_MAX_TOKENS` limits each generated capsule, not the source file.
`CM_SEMANTIC_INPUT_CHUNK_CHARS` is an approximate input-size guard chosen to
remain comfortable within an 8192-token model context. `CM_SEMANTIC_MERGE_FAN_IN`
controls how many lower-level capsules are combined per hierarchy request.

## Interactive memory-store console

The MCP server starts a small localhost-only TCP console by default. It changes
which stores are searched when `recall_memory` omits its `stores` argument.
Explicit tool arguments still take precedence.

Start the MCP server normally, then connect from another terminal:

```bash
nc 127.0.0.1 8765
```

Available commands:

```text
stores
use prompts
use main prompts
use all
status
help
exit
```

`use all` restores federated retrieval across every enabled store. Configure or
disable the console in `.env`:

```dotenv
CM_CONSOLE_ENABLED=true
CM_CONSOLE_HOST=127.0.0.1
CM_CONSOLE_PORT=8765
```

### Console without a TCP port (recommended with LM Studio)

LM Studio can keep the MCP process and its stderr hidden. The separate console
command writes the active-store selection into the same `CM_DATA_DIR` used by
the server, and the server reloads it on every recall:

```bash
contextual-memory-console
memory> stores
memory> use prompts
memory> status
```

A one-shot form is also available:

```bash
contextual-memory-console use prompts
contextual-memory-console status
```

No `nc` connection or listening port is required. Both LM Studio and the
console command must use the same `CM_DATA_DIR`.

### Media ingestion concurrency

Image analysis, audio transcription/distillation, and sampled video-frame analysis can use bounded concurrency:

```env
CM_MEDIA_CONCURRENCY=1  # sequential; use 2+ for bounded parallel media requests
```

A one-off scan can override it with `--media-concurrency N`. Images and audio files are processed concurrently across a directory scan. Video files remain sequential at the file level, while each video's sampled frames are analyzed concurrently, preventing concurrency from multiplying across multiple videos.

## Guaranteed automatic chat capture (two-pass proxy)

MCP servers do not automatically receive the user's raw chat messages. They only
see tool calls that the host model elects to make, so MCP instructions alone
cannot guarantee that every durable statement is stored.

This project includes an optional OpenAI-compatible chat proxy that observes the
actual request and response. It forwards the normal completion to LM Studio and
then runs a dedicated memory-extraction pass. The extraction pass writes direct
user statements to the configured writable memory store without requiring the
chat model to call `store_memory`.

Start the proxy:

```bash
uv run contextual-memory-chat-proxy
```

Then change the chat application's OpenAI-compatible base URL from LM Studio's
usual `http://127.0.0.1:1234/v1` to:

```text
http://127.0.0.1:1235/v1
```

### Same model, two passes

This mode sends the normal reply first, then asks the same requested model to
extract durable memories from the latest exchange:

```env
CM_CHAT_CAPTURE_MODE=same-model
CM_CHAT_UPSTREAM_BASE_URL=http://127.0.0.1:1234/v1
CM_CHAT_CAPTURE_STORE=main
```

Only one model needs to be loaded. It performs two sequential inference calls.
The second pass runs after the response has been returned by the upstream model
and is scheduled independently from the client response.

### Separate extractor model

This mode keeps the main chat model and memory extractor independently
configurable:

```env
CM_CHAT_CAPTURE_MODE=separate-model
CM_CHAT_UPSTREAM_BASE_URL=http://127.0.0.1:1234/v1
CM_CHAT_CAPTURE_BASE_URL=http://127.0.0.1:1234/v1
CM_CHAT_CAPTURE_MODEL=qwen2.5-7b-instruct
CM_CHAT_CAPTURE_STORE=main
```

`CM_CHAT_CAPTURE_BASE_URL` may point to a second LM Studio/Ollama/OpenAI-compatible
server on another machine or port. When it is omitted, the extractor uses the
same upstream endpoint but requests `CM_CHAT_CAPTURE_MODEL`.

### Verify that capture is actually happening

The proxy exposes a local diagnostics endpoint:

```text
http://127.0.0.1:1235/v1/contextual-memory/status
```

Its counters distinguish the common failure cases:

- `observed_requests`: requests that reached a capture-aware endpoint.
- `scheduled_passes`: extraction jobs accepted.
- `completed_passes`: successful extractor responses.
- `stored_memories`: memories actually written.
- `failed_passes` and `last_error`: endpoint, JSON, or storage failures.
- `last_endpoint`: either `/v1/chat/completions` or `/v1/responses`.

The proxy captures both OpenAI-compatible APIs. Earlier builds only captured
`/v1/chat/completions`, so clients using `/v1/responses` silently bypassed
automatic retention.

Available settings:

- `CM_CHAT_CAPTURE_MODE`: `off`, `same-model`, or `separate-model`.
- `CM_CHAT_CAPTURE_STORE`: explicit writable destination store; defaults to `main`.
- `CM_CHAT_CAPTURE_MODEL`: required only for `separate-model`.
- `CM_CHAT_CAPTURE_BASE_URL`: optional extractor endpoint.
- `CM_CHAT_CAPTURE_MAX_MEMORIES`: maximum atomic memories stored per exchange.
- `CM_CHAT_CAPTURE_MAX_TOKENS`: extractor output budget.
- `CM_CHAT_CAPTURE_TIMEOUT`: extractor request timeout.

The proxy currently buffers streamed upstream responses before relaying them, so
clients receive valid streaming data but not token-by-token low-latency delivery.
Use non-streaming chat completion requests when response latency matters most.