# agent-memory

> **Long-term memory for agents, done properly.** A memory service that keeps
> per-user memories, ranks recall by relevance, recency, and importance together,
> consolidates repeated events into semantic facts, and forgets low-value memories
> over time. Retrieval is always scoped, so one user's memory can never surface for
> another. Exposed as an MCP server so an agent can remember and recall over the
> protocol. Fully offline and keyless.

[![ci](https://github.com/tahasiddiquii/agent-memory/actions/workflows/ci.yml/badge.svg)](https://github.com/tahasiddiquii/agent-memory/actions/workflows/ci.yml)
![python](https://img.shields.io/badge/python-3.11%2B-blue)
![license](https://img.shields.io/badge/license-MIT-green)
![offline](https://img.shields.io/badge/runs-offline%20%C2%B7%20zero--key-success)

An agent that forgets everything between sessions cannot help you twice. But naive
memory is worse than none: dump every message into a vector store and recall
returns the most recent chatter instead of the fact you need, or worse, one user's
data leaks into another's session. This service treats memory as a ranking and
governance problem, built from my retrieval and evaluation work.

## What this demonstrates

| Capability | Where |
| --- | --- |
| Scoped memory: retrieval never crosses users | [memory.py](src/agent_memory/memory.py) |
| Ranking by relevance, recency, and importance together | [memory.py](src/agent_memory/memory.py) |
| Grounded recall with a refusal path | [recall.py](src/agent_memory/recall.py) |
| Consolidation of repeats into semantic facts | [memory.py](src/agent_memory/memory.py) |
| Decay and forgetting of low-value memories | [memory.py](src/agent_memory/memory.py) |
| Exposed as an MCP server | [server.py](src/agent_memory/server.py) |
| Recall quality and isolation gated in CI | [evals.py](src/agent_memory/evals.py) |

## Architecture

```mermaid
flowchart LR
    E[events per user] --> S[(scoped store)]
    S --> C[consolidate repeats to facts]
    S --> D[decay and forget low value]
    Q[question + user scope] --> R{{relevance + recency + importance}}
    S --> R
    R --> A[grounded recall or refuse]
```

## Quickstart

```bash
make dev            # venv + install -e ".[dev]"

amem demo           # ranked recall vs a most-recent baseline, plus consolidation and decay
amem recall "what is my favorite programming language" --scope alice
amem eval           # the recall and isolation gate
amem serve          # live MCP server: remember / recall / consolidate
```

No keys, no network. Embeddings are a deterministic hashing vectorizer; set a real
embedder behind the same interface in production.

## The gate that matters

`amem eval` answers questions whose facts were introduced in earlier sessions
([report](reports/memory_report_example.md)):

| metric | value | gate |
| --- | --- | --- |
| **full_recall** | **1.000** | >= 0.90 |
| naive_recall (most-recent) | 0.200 | < full |
| precision_at_k | 1.000 | >= 0.80 |
| **cross_scope_leaks** | **0** | = 0 |
| consolidation_merged | 2 | reported |
| forgotten_after_decay | 3 | reported |

The comparison is the point. Ranked recall answers every question; a most-recent
baseline answers one in five, because the fact you asked about is usually an older
memory buried under recent chatter. Scoped retrieval leaks nothing across users
even though two users have a memory about the same topic. CI fails if recall drops,
if the baseline is not beaten, or if a single cross-scope leak appears.

## What it does

- **Ranks, not just stores.** Recall combines relevance, recency, and importance,
  so a salient old fact beats a trivial recent one. Naive most-recent recall gets
  this wrong, which the eval measures directly.
- **Keeps users apart.** Two users each say "my favorite language is ..."; each only
  ever recalls their own. Isolation is structural, not best-effort.
- **Consolidates.** Repeated events are folded into a single semantic fact with
  boosted importance, so the store does not bloat with duplicates.
- **Forgets on purpose.** Low-importance, old memories decay below a threshold and
  are pruned, while important facts survive. Forgetting is a feature, and the eval
  reports what was dropped.
- **Refuses.** With nothing relevant in scope, recall says it does not remember
  rather than returning noise.

## Design decisions

- **Scope is the first filter.** Retrieval is restricted to the caller's scope
  before ranking, so cross-user leakage is impossible by construction.
- **One score, three signals.** Relevance alone recalls stale facts; recency alone
  recalls chatter; importance alone ignores the query. Combining them is what makes
  recall useful.
- **Decay is query-free.** Forgetting uses recency and importance only, so the store
  sheds low-value memories independent of any particular question.

## Layout

```
src/agent_memory/  embed · memory · recall · server · evals · cli
data/  memories.jsonl · questions.jsonl
reports/  memory_report_example.md
```

## Related repositories

Part of a portfolio on production ML and LLM engineering:

- [hybrid-graph-rag](https://github.com/tahasiddiquii/hybrid-graph-rag): hybrid and graph retrieval benchmark
- [mcp-knowledge-server](https://github.com/tahasiddiquii/mcp-knowledge-server): permission-aware knowledge MCP server
- [llm-eval-observability](https://github.com/tahasiddiquii/llm-eval-observability): RAG evaluation and observability
- [ai-harness](https://github.com/tahasiddiquii/ai-harness): multi-stage agent harness
- **agent-memory**: this repo.

## License

MIT (c) 2026 Taha Siddiqui