MCP Memory Server

A semantic memory layer for AI coding agents. It allows agents to store and retrieve memories with automatic summarization, reducing token usage during long coding sessions.

Features

• Automatic Summarization — Stores the original text and generates a concise summary in the background (Groq primary, Anthropic fallback).

• Semantic Retrieval — Uses vector embeddings based on the raw text for accurate relevance matching. Returns summaries by default to save tokens; full text is opt-in via include_full_text.

• Recency-Weighted Reranking — Retrieval blends vector similarity with a recency score so recent memories get a slight ranking boost over older ones with similar content.

• Near-Duplicate Detection — Storing a memory that closely matches an existing one updates the existing record instead of creating a duplicate.

• Non-Blocking Writes — Store calls return immediately after embedding; summarization runs asynchronously and populates a few seconds later.

• MCP Native Support — Exposes store_memory, retrieve_memory, and delete_memory as MCP tools.

• REST API — Simple HTTP endpoints for testing and integration (/store, /retrieve, /health).

• Session & Project Support — Optional session_id and project_id fields for organizing memories.

• Efficient — Uses ONNX backend with quantized embeddings for lower memory usage.

Related MCP server: umo-memory

Quick Start

pip install -r requirements.txt
python -m memory_server

That's it — no database to run. Memories live in a single SQLite file at ~/.memory_server/memories.db (override with MEMORY_DB_PATH). The server runs at http://localhost:8000.

The original Weaviate backend is still available:

pip install weaviate-client
docker compose up -d
MEMORY_BACKEND=weaviate python -m memory_server

Interactive Documentation

Open http://localhost:8000/docs in your browser to test the API.

Connect from an MCP client

The server exposes MCP over streamable HTTP at http://localhost:8000/mcp — any MCP-compatible agent can use it.

Claude Code

claude mcp add memory --transport http http://localhost:8000/mcp

Hermes Agent (or any other MCP client) — add an MCP server entry pointing at http://localhost:8000/mcp in your client's MCP configuration.

Agents get three tools: store_memory, retrieve_memory, delete_memory — with summaries returned by default so retrieval stays token-cheap.

Usage

REST Endpoints

Store a memory

curl -X POST http://localhost:8000/store \
  -H "Content-Type: application/json" \
  -d '{
    "text": "The agent is building a memory layer using Weaviate and FastAPI.",
    "source": "conversation",
    "category": "project",
    "session_id": "session-123",
    "project_id": "mcp-memory"
  }'

Retrieve memories (with optional filters)

curl -X POST http://localhost:8000/retrieve \
  -H "Content-Type: application/json" \
  -d '{
    "query": "memory layer",
    "top_k": 5,
    "session_id": "session-123",
    "category": "project",
    "source": "conversation",
    "include_full_text": false
  }'

Note: Retrieval returns summaries and metadata by default. Set "include_full_text": true to include the original text in results. This keeps token usage low when only the summary is needed.

Delete a memory

curl -X DELETE http://localhost:8000/memory/<memory-id>

MCP Tools

The server exposes tools via the Model Context Protocol at:

http://localhost:8000/mcp

Available tools:

• store_memory(text, source, category, tags, session_id, project_id) — Returns immediately; summarization runs in the background. The summary field on a freshly stored memory may be empty for a few seconds until the background task completes.

• retrieve_memory(query, top_k, session_id, project_id, category, source, include_full_text) — Returns summary + metadata by default. Pass include_full_text=true to include original text.

• delete_memory(memory_id)

These can be called directly by any MCP-compatible agent.

Behavior Notes

• Async summarization: Both store_memory and POST /store return as soon as the embedding is computed and the record is inserted. The LLM-generated summary is populated asynchronously a few seconds later. If you retrieve a memory immediately after storing it, the summary field may be empty.

• Near-duplicate merging: If a new memory's embedding is within a cosine distance of 0.05 of an existing record, the existing record is updated rather than creating a new one. This prevents redundant entries when the same fact is stored with minor wording differences.

• Recency reranking: Retrieved results are reranked using a combined score of vector similarity and recency. Recent memories receive a slight boost. The decay weight is small enough (0.01 per day) that strong semantic matches still outrank recent but less relevant ones.

Environment Variables

Billing note for ANTHROPIC_API_KEY: Each deployer brings their own Anthropic key. The project does not supply or cover Anthropic API usage on anyone's behalf — whoever sets that env var is the one whose account gets billed if the Groq path fails and the Anthropic fallback fires.

Summarization follows a three-step fallback chain: Groq first (free), Anthropic only if Groq fails (paid, deployer's own key), plain truncation to 600 characters if both fail.

Architecture

• Storage: SQLite, single file, zero infrastructure (default) — brute-force cosine search over float32 blobs, sub-millisecond at this scale. Weaviate available as a legacy backend.

• Embeddings: sentence-transformers/all-MiniLM-L6-v2 (ONNX backend), computed from raw text

• Summarization: Groq (Llama 3.1 8B, primary) / Anthropic Claude (paid fallback), runs asynchronously after insert

• Framework: FastAPI + FastMCP

Tunable constants (hardcoded in sqlite_store.py/store.py, not env vars):

Project Structure

mcp-memory-server/
├── memory_server/
│   ├── __init__.py
│   ├── main.py
│   ├── sqlite_store.py   # default backend (zero infra)
│   ├── store.py          # legacy Weaviate backend
│   └── summarize.py
├── docker-compose.yml    # only needed for the Weaviate backend
├── requirements.txt
├── diagnose.py
├── test_e2e.py
├── eval_retrieval.py
└── README.md

License

MIT License