Contextual Memory MCP

A local-first persistent contextual-memory engine with an MCP adapter for AI models such as Qwen.

The project is being developed incrementally. The first ingestion target is a directory of reusable Markdown prompt material, but the storage and retrieval layers are general-purpose.

Architecture

Text, images, audio, and video
          |
          v
Modality-specific semantic distillation
          |
          +--> SQLite semantic capsules and sparse concepts
          |
          +--> packed int8 latent parameter bank
          |
          v
Hybrid retrieval and context assembly
          |
          v
Thin MCP adapter used by Qwen

The CLI performs administrative work. The MCP server exposes memory operations that are useful to the model.

Related MCP server: contextgit

Install with uv

uv venv --python 3.12
uv pip install -e .
cp .env.example .env

Python 3.12 or 3.13 is recommended for broad compatibility with the current machine-learning dependencies.

Index files and directories

The CLI accepts either one file or an entire directory. With no --name, the content is added to the normal writable main store:

uv run contextual-memory-index scan /path/to/character.md
uv run contextual-memory-index scan /path/to/saved/prompts

Provide --name to add the target to a named database. If that database does not exist, it is created automatically:

uv run contextual-memory-index scan /path/to/character.md --name roleplay

New named databases are immutable after indexing by default. Use --mutable when the named database should remain writable:

uv run contextual-memory-index scan /path/to/saved/prompts --name prompt-library --mutable

A later explicit scan can update an existing named database. Locked databases are reopened only for that update and then returned to their previous mode. Use --replace when the entire named database should be rebuilt instead:

uv run contextual-memory-index scan /path/to/saved/prompts --name prompt-library --replace

Directory scans support exclusions, and either kind of scan can force unchanged content to be indexed again:

uv run contextual-memory-index scan /path/to/saved/prompts --exclude archive --force

Clear stored data

Interactive confirmation:

uv run contextual-memory-index clear --store main

Non-interactive confirmation:

uv run contextual-memory-index clear --store main --yes

Scanning and clearing are deliberately CLI-only maintenance operations. They are not exposed to the model through MCP.

Run the MCP server

uv run contextual-memory-mcp

The MCP server exposes model-facing recall, ingestion, lifecycle, and maintenance tools:

recall_memory

Retrieves semantically, lexically, and graph-related memory and returns clean, source-attributed Markdown that Qwen can use in its active context.

Typical model call:

recall_memory(
  query="Create a gender-swapped character reference sheet while preserving identity, colors, and species traits"
)

store_memory

Automatically captures durable information directly stated by the user. The model is instructed to call it before drafting its response whenever the current message contains durable information, including during ordinary conversation. The user does not need to say “remember this,” and routine storage must not trigger a permission question or an announcement. The model supplies an integer memory type and a normalized importance score from 0.0 to 2.0. The server validates those values and assigns lifecycle state, origin, confidence, source quality, and conservative server-owned lifecycle policy.

store_memory_candidate

Stores model-generated hypotheses and interpretations as candidate inferences. It is not a substitute for store_memory when the user directly stated the information, whenever that statement is durable. The model supplies semantic type and importance, while the server assigns lifecycle state, origin, confidence, source quality, and conservative retention policy.

explore_memory

Traverses the knowledge graph around a concept and returns related concepts plus supporting memory excerpts.

The MCP does not expose scan, clear, stats, or database-deletion tools. Those are administrative concerns and remain under direct user control.

LM Studio configuration

Add the server as a stdio MCP using the repository's virtual environment. A typical configuration resembles:

{
  "mcpServers": {
    "contextual-memory": {
      "command": "/absolute/path/to/contextual-memory-mcp/.venv/bin/contextual-memory-mcp",
      "args": [],
      "cwd": "/absolute/path/to/contextual-memory-mcp"
    }
  }
}

Alternatively, invoke it through uv:

{
  "mcpServers": {
    "contextual-memory": {
      "command": "uv",
      "args": [
        "run",
        "--directory",
        "/absolute/path/to/contextual-memory-mcp",
        "contextual-memory-mcp"
      ]
    }
  }
}

Use absolute paths when LM Studio is launched outside a terminal, because GUI applications may not inherit the same shell PATH.

Storage

Persistent data is stored beneath CM_DATA_DIR, which defaults to ./data:

• contextual_memory.sqlite3

• vectors/latent_vectors.sqlite3(compact int8 parameter bank)

• store_overlays.sqlite3 (local ranking overlays for immutable stores)

• stores/<name>/manifest.json plus each scanned store database

The embedding model is stored in the application's own model directory and normal server/index runs load it with network access disabled. Install it once before the first scan or recall:

uv run contextual-memory-index model install

Check the local installation without contacting Hugging Face:

uv run contextual-memory-index model status

The default location is CM_DATA_DIR/models. Override it with CM_MODELS_DIR. After installation, deleting the global Hugging Face cache does not affect this project. A missing model produces an explicit error instead of downloading during an MCP or indexing operation.

Current retrieval path

recall_memory currently combines:

• semantic similarity from the packed int8 parameter bank

• SQLite full-text relevance

• graph relationships

• segment importance

• per-source diversity

• a configurable context-size budget

The returned value is Markdown rather than an embedding blob or database record, so the model receives reusable source material directly.

Memory importance and lifecycle

Memory importance is persistent and evolves through an explicit maintenance pass:

new recall accesses
      +
configured access gain
      -
elapsed inactivity decay
      |
      v
updated importance
      |
      v
promotion / archival policy

Run run_memory_maintenance through MCP to reinforce accessed memories, decay inactive unpinned memories, synchronize vector metadata, and then evaluate candidate promotion or active-memory archival. Pass dry_run=true to inspect all proposed changes without mutating storage.

All state, type, origin, lifecycle-reason, and importance-reason enums are stored as SQLite integers and exposed in Python as IntEnum values.

Multiple memory stores

The server provides a writable main store. Scanned databases live beneath data/stores/<name>/ and contain their own manifest.json. The filesystem is the source of truth: there is no separate store registry, registration step, mount operation, or startup fixture.

list_memory_stores discovers manifests directly from disk. A named store is opened lazily when recall or another operation addresses it. Returned memory IDs are globally qualified, for example ghidra-core:seg_abcd.

Store modes are SQLite integers exposed through StoreMode:

0 READ_WRITE
1 READ_ONLY
2 IMMUTABLE

Writes require an explicit target_store; they are never broadcast. Before a write, clients must call list_memory_stores whenever the destination is not already established in the current context. Store IDs must not be guessed or discovered by intentionally causing a failed write.

Maintenance runs only against writable stores. Read-only and immutable SQLite databases are opened with SQLite read-only URI modes and are never migrated automatically. Their recall access counts and user-specific ranking choices are kept in store_overlays.sqlite3 instead of changing canonical content.

Creating a scanned database is sufficient to make it discoverable:

uv run contextual-memory-index scan ./project --name project-memory

This creates:

data/stores/project-memory/
├── manifest.json
├── project-memory.sqlite3
└── vectors/

No registration or mounting command is required.

Web search providers

Automatic web acquisition can use multiple search providers in priority order. Providers that need credentials are skipped unless configured, and failures automatically fall through to the next provider.

CM_WEB_SEARCH_PROVIDERS=exa,brave,tavily,searxng,duckduckgo
CM_WEB_SEARCH_TIMEOUT=12
CM_EXA_API_KEY=
CM_BRAVE_SEARCH_API_KEY=
CM_TAVILY_API_KEY=
CM_SEARXNG_URL=http://localhost:8080

With no API keys or SearXNG URL, DuckDuckGo is used as the zero-configuration fallback. For a self-hosted setup, place searxng first and set CM_SEARXNG_URL. The configured SearXNG instance must permit JSON search responses.

Web acquisition and indexing

By default, indexed memory is authoritative over the web. recall_memory only triggers web discovery, page ingestion, and a second local recall when local retrieval returns no hits. Search providers are attempted in configured priority order; unconfigured providers are skipped and failures fall through to the next provider.

CM_MEMORY_AUTHORITY_OVER_WEB=true
CM_WEB_SEARCH_PROVIDERS=exa,brave,tavily,searxng,duckduckgo
CM_WEB_SEARCH_CACHE_DAYS=7
CM_WEB_ACQUISITION_RETRY_DAYS=7
CM_WEB_ACQUISITION_REFRESH_DAYS=90

Search results are cached in data/web_acquisition.sqlite3. Acquisition history suppresses repeated searches for unavailable topics and schedules successful topics for a later refresh.

The index CLI accepts directories, individual files, direct URLs, and search queries:

python main.py scan ./prompts
python main.py scan ./character.md --name roleplay
python main.py scan https://example.org/wiki/Character --name roleplay
python main.py scan --search "Character franchise personality dialogue" --name roleplay

When --name is omitted, content is indexed into the writable main store. Missing named stores are created automatically and are immutable after indexing unless --mutable is supplied. MediaWiki pages use their API when available, and GitHub blob URLs are fetched through their raw-content endpoint.

Visual memory

scan now discovers common image formats and sends them to an OpenAI-compatible vision model. LM Studio's local server is the default endpoint.

contextual-memory-index scan ./reference-images \
  --name visual-reference \
  --vision-model qwen2.5-vl-7b-instruct

The extractor stores a searchable visual summary plus independently classified text regions. Its general-purpose taxonomy distinguishes speech and thought bubbles, captions/overlays, clothing text, text attached to physical surfaces, documents, UI/HUD text, sound effects, watermarks, and uncategorized text. Every region includes normalized bounding coordinates and separate OCR and classification confidence values. This lets retrieval distinguish, for example, a caption saying "red coat" from the words printed on the coat itself.

Configuration:

• CM_VISION_MODEL: default vision model name (required when images are found)

• CM_VISION_BASE_URL: compatible /v1 endpoint; defaults to LM Studio at http://127.0.0.1:1234/v1

• CM_VISION_API_KEY: optional bearer token

• CM_VISION_TIMEOUT: request timeout in seconds

• CM_VISUAL_GLOBS: comma-separated image globs

Use --no-vision to skip image files during a directory scan. Scanning a single image with --no-vision is rejected rather than silently doing nothing.

Latent multimodal memory

This branch stores learned external-memory representations rather than copies of source training material. The scanner reads a source only during ingestion, distills it into a lossy semantic capsule, embeds that capsule, quantizes the normalized embedding to signed int8, and persists the compact parameter record. Raw image, audio, and video bytes are never copied into the memory store.

Supported inputs:

• Text: works immediately through the built-in heuristic semantic provider. Configure CM_SEMANTIC_MODEL later for higher-quality local-LLM distillation.

• Images: analyzed only when CM_VISION_MODEL is configured, including classified text regions. Otherwise directory scans skip images and report the count.

• Audio: transcribed only when CM_AUDIO_MODEL is configured, then semantically distilled. Otherwise directory scans skip audio and report the count.

• Video: enabled only when both vision and audio providers are configured. It is sampled with ffmpeg, frames are analyzed, audio is transcribed, and both channels become a compact temporal capsule.

Optional local services are enabled per modality:

CM_SEMANTIC_PROVIDER=auto
# CM_SEMANTIC_MODEL=your-local-instruct-model
# CM_VISION_MODEL=your-local-vision-model
# CM_AUDIO_MODEL=your-local-whisper-model

Configured endpoints use the OpenAI-compatible API and default to LM Studio's local http://127.0.0.1:1234/v1. Unconfigured modalities do not prevent the server or text scanner from running. Audio/video ingestion requires ffmpeg on PATH only when those modalities are actually processed. The original wording or media cannot be reconstructed from the stored capsules and quantized vectors; retain source files separately when exact reproduction or citation is required.

Parameter storage

The former vector collection has been replaced by latent_vectors.sqlite3. Each row contains only a stable memory ID, dimensions, a scalar quantization scale, and a packed int8 vector. Retrieval uses exact integer-vector scanning and SQLite remains authoritative for semantic capsules, concept weights, and lifecycle state.

Hierarchical latent distillation

Large text files are distilled hierarchically rather than sent to the semantic model in one request. Source text is split into bounded input chunks, chunk capsules are merged into section capsules, and section capsules are merged into a document capsule. The stored latent document contains the compact document capsule and section capsules; original source wording and transient leaf capsules are not retained.

Configuration:

CM_SEMANTIC_MAX_TOKENS=512
CM_SEMANTIC_INPUT_CHUNK_CHARS=12000
CM_SEMANTIC_MERGE_FAN_IN=6
CM_SEMANTIC_CONCURRENCY=1  # 1 = sequential; 2+ enables bounded parallel requests

CM_SEMANTIC_MAX_TOKENS limits each generated capsule, not the source file. CM_SEMANTIC_INPUT_CHUNK_CHARS is an approximate input-size guard chosen to remain comfortable within an 8192-token model context. CM_SEMANTIC_MERGE_FAN_IN controls how many lower-level capsules are combined per hierarchy request.

Interactive memory-store console

The MCP server starts a small localhost-only TCP console by default. It changes which stores are searched when recall_memory omits its stores argument. Explicit tool arguments still take precedence.

Start the MCP server normally, then connect from another terminal:

nc 127.0.0.1 8765

Available commands:

stores
use prompts
use main prompts
use all
status
help
exit

use all restores federated retrieval across every enabled store. Configure or disable the console in .env:

CM_CONSOLE_ENABLED=true
CM_CONSOLE_HOST=127.0.0.1
CM_CONSOLE_PORT=8765

Console without a TCP port (recommended with LM Studio)

LM Studio can keep the MCP process and its stderr hidden. The separate console command writes the active-store selection into the same CM_DATA_DIR used by the server, and the server reloads it on every recall:

contextual-memory-console
memory> stores
memory> use prompts
memory> status

A one-shot form is also available:

contextual-memory-console use prompts
contextual-memory-console status

No nc connection or listening port is required. Both LM Studio and the console command must use the same CM_DATA_DIR.

Media ingestion concurrency

Image analysis, audio transcription/distillation, and sampled video-frame analysis can use bounded concurrency:

CM_MEDIA_CONCURRENCY=1  # sequential; use 2+ for bounded parallel media requests

A one-off scan can override it with --media-concurrency N. Images and audio files are processed concurrently across a directory scan. Video files remain sequential at the file level, while each video's sampled frames are analyzed concurrently, preventing concurrency from multiplying across multiple videos.

Guaranteed automatic chat capture (two-pass proxy)

MCP servers do not automatically receive the user's raw chat messages. They only see tool calls that the host model elects to make, so MCP instructions alone cannot guarantee that every durable statement is stored.

This project includes an optional OpenAI-compatible chat proxy that observes the actual request and response. It forwards the normal completion to LM Studio and then runs a dedicated memory-extraction pass. The extraction pass writes direct user statements to the configured writable memory store without requiring the chat model to call store_memory.

Start the proxy:

uv run contextual-memory-chat-proxy

Then change the chat application's OpenAI-compatible base URL from LM Studio's usual http://127.0.0.1:1234/v1 to:

http://127.0.0.1:1235/v1

Same model, two passes

This mode sends the normal reply first, then asks the same requested model to extract durable memories from the latest exchange:

CM_CHAT_CAPTURE_MODE=same-model
CM_CHAT_UPSTREAM_BASE_URL=http://127.0.0.1:1234/v1
CM_CHAT_CAPTURE_STORE=main

Only one model needs to be loaded. It performs two sequential inference calls. The second pass runs after the response has been returned by the upstream model and is scheduled independently from the client response.

Separate extractor model

This mode keeps the main chat model and memory extractor independently configurable:

CM_CHAT_CAPTURE_MODE=separate-model
CM_CHAT_UPSTREAM_BASE_URL=http://127.0.0.1:1234/v1
CM_CHAT_CAPTURE_BASE_URL=http://127.0.0.1:1234/v1
CM_CHAT_CAPTURE_MODEL=qwen2.5-7b-instruct
CM_CHAT_CAPTURE_STORE=main

CM_CHAT_CAPTURE_BASE_URL may point to a second LM Studio/Ollama/OpenAI-compatible server on another machine or port. When it is omitted, the extractor uses the same upstream endpoint but requests CM_CHAT_CAPTURE_MODEL.

Verify that capture is actually happening

The proxy exposes a local diagnostics endpoint:

http://127.0.0.1:1235/v1/contextual-memory/status

Its counters distinguish the common failure cases:

• observed_requests: requests that reached a capture-aware endpoint.

• scheduled_passes: extraction jobs accepted.

• completed_passes: successful extractor responses.

• stored_memories: memories actually written.

• failed_passes and last_error: endpoint, JSON, or storage failures.

• last_endpoint: either /v1/chat/completions or /v1/responses.

The proxy captures both OpenAI-compatible APIs. Earlier builds only captured /v1/chat/completions, so clients using /v1/responses silently bypassed automatic retention.

Available settings:

• CM_CHAT_CAPTURE_MODE: off, same-model, or separate-model.

• CM_CHAT_CAPTURE_STORE: explicit writable destination store; defaults to main.

• CM_CHAT_CAPTURE_MODEL: required only for separate-model.

• CM_CHAT_CAPTURE_BASE_URL: optional extractor endpoint.

• CM_CHAT_CAPTURE_MAX_MEMORIES: maximum atomic memories stored per exchange.

• CM_CHAT_CAPTURE_MAX_TOKENS: extractor output budget.

• CM_CHAT_CAPTURE_TIMEOUT: extractor request timeout.

The proxy currently buffers streamed upstream responses before relaying them, so clients receive valid streaming data but not token-by-token low-latency delivery. Use non-streaming chat completion requests when response latency matters most.