s3-agent-memory

Long-term memory for AI agents stored as plain JSON objects in any S3-compatible bucket: no database, no vector DB service. Keyword + optional local-embedding search, safe concurrent writes with S3 conditional requests, an MCP server and a LangGraph store.

s3-agent-memory gives an agent typed memories (semantic facts, episodic events, procedural instructions and preferences) that survive across conversations. Each memory is one JSON object at <prefix>/<namespace>/<kind>/<id>.json, next to a small per-namespace BM25 index and, optionally, a compact numpy shard of local embeddings. It works on AWS S3, Cloudflare R2, SeaweedFS, MinIO, a local directory or an in-process dict. Writes use S3 conditional requests (If-None-Match: *, If-Match: <etag>), so many agents can write to the same namespace without losing updates. Runs 100% locally, no paid APIs required.

Why

Most agent-memory stacks need a database and a vector service before the first fact is stored. For a few thousand memories per user that is a lot of infrastructure: an S3 bucket already gives durability, versioning, encryption, lifecycle rules, IAM and a price close to zero. What it lacks is search and safe concurrent updates, and that is what this library adds:

• Search without a service. A per-namespace index (term frequencies and ranking metadata, never the text) makes recall one GET plus the top-k objects. BM25 is pure Python; embeddings are an opt-in extra that runs a local open-source model (fastembed / sentence-transformers).

• Concurrency without locks. Every shared object is changed with compare-and-swap on its ETag, so a lost race is retried, not lost. Tested with threads against moto, the filesystem backend and a real SeaweedFS server in CI.

• Isolation by construction. Namespaces and ids go through a strict allow-list; the MCP server is bound to one namespace and has no namespace argument; an object found under the wrong key is refused.

• Plain files. Memories are readable JSON. Export/import is JSONL, and the index can always be rebuilt from the objects.

Related MCP server: MemoryGate

Quickstart

pip install "s3-agent-memory[s3] @ git+https://github.com/MartinCastroAlvarez/s3-agent-memory"
s3-agent-memory remember "Alice prefers metric units" --kind procedural -n users/alice
s3-agent-memory recall "which units?" -n users/alice          # -> JSON with score + provenance
claude mcp add memory -- s3-agent-memory serve -n users/alice  # MCP for Claude Code
docker compose up -d && python examples/s3_demo.py             # same thing on local S3

Without --url the CLI stores memories in ./.agent-memory/ (filesystem backend). It will be pip install "s3-agent-memory[s3]" once the first release is on PyPI.

from s3_agent_memory import MemoryStore

store = MemoryStore.from_url("s3://my-bucket/agents")   # or file://./mem, memory://
store.remember("users/alice", "Alice is vegetarian", tags=["food"], importance=0.8)
store.remember("users/alice", "Asked for a ramen recipe", kind="episodic", ttl=7 * 86400)
for hit in store.recall("users/alice", "what can Alice eat?"):
    print(hit.score, hit.memory.content, hit.key)

How it works

flowchart LR
    subgraph Clients
        MCP["MCP server (stdio)<br/>remember · recall · forget · list"]
        LG["LangGraph BaseStore<br/>S3Store"]
        API["Python API · CLI"]
    end
    MCP & LG & API --> MS["MemoryStore<br/>validate · limits · CAS retries"]
    MS -->|"PUT If-None-Match: *"| OBJ[("&lt;ns&gt;/&lt;kind&gt;/&lt;id&gt;.json<br/>source of truth")]
    MS -->|"PUT If-Match: etag"| IDX[("&lt;ns&gt;/_index.json<br/>term freqs + metadata")]
    MS -.->|optional| VEC[("&lt;ns&gt;/_vectors.npz<br/>float16, local model")]
    MS -->|forget| TS[("&lt;ns&gt;/_tombstones/&lt;id&gt;.json")]
    IDX --> R["recall: BM25 (+ cosine)<br/>+ recency + importance"]
    VEC -.-> R
    R -->|"top-k GETs"| OBJ
    subgraph Backends
        S3["AWS S3 · R2 · SeaweedFS · MinIO"]
        FS["local directory"]
        MEM["in-memory"]
    end

1. remember reserves the id in the index (compare-and-swap), then creates the memory object with If-None-Match: *, then (if an embedder is set) adds its vector to the shard, again with compare-and-swap. Ids are unique per namespace across kinds.

2. recall reads the index, scores every eligible memory (not expired, kind and tag filters) and fetches only the winners: score = 0.7 * relevance + 0.15 * recency + 0.15 * importance, where relevance is BM25 normalised to the best hit (mixed 50/50 with cosine similarity when embeddings exist), recency is 0.5 ** (age / half_life) with half-lives of 30 days for episodic, 180 for semantic and none for procedural memories, and importance is the 0-1 prior you gave. An empty query ranks by recency and importance. Weights live in Ranking.

3. update writes with If-Match on the memory's ETag and bumps its version. Pass expected_version= for strict optimistic concurrency (a concurrent change raises ConflictError); without it a lost race re-reads and re-applies the change.

4. forget writes a tombstone (id, kind, time, reason; no content), hard-deletes the object, and removes it from the index and shard. prune_expired forgets memories past their TTL.

5. consolidate groups near-duplicates per kind (token Jaccard, plus cosine when embeddings exist), merges each group with a summariser (a local ollama model by default, LongestSummariser with no model), records consolidated_from, and forgets the originals.

6. export / import move memories as JSONL. Import writes only into the target namespace (the namespace in the file is ignored), skips existing ids unless overwrite=True, and never resurrects a forgotten id unless resurrect=True.

7. reindex rebuilds the index and shard from the objects, so the index is a cache you can delete.

Object layout

s3://bucket/<prefix>/users/alice/semantic/01a16b209bf83fe5ba69f185.json
s3://bucket/<prefix>/users/alice/episodic/01a16b20a1c2d3e4f5a6b7c8.json
s3://bucket/<prefix>/users/alice/procedural/01a16b209bf8ad13a72e486e.json
s3://bucket/<prefix>/users/alice/_index.json
s3://bucket/<prefix>/users/alice/_vectors.npz          # only with an embedder
s3://bucket/<prefix>/users/alice/_tombstones/<id>.json

{"schema":1,"id":"01a16b209bf8ad13a72e486e","namespace":["users","alice"],"kind":"procedural",
 "content":"Alice prefers metric units","source":"chat:42","importance":0.9,"tags":["prefs"],
 "created_at":"2026-10-23T23:37:41.624370Z","updated_at":"2026-10-23T23:37:41.624370Z",
 "expires_at":null,"data":null,"metadata":{},"version":1}

Ids are time-ordered (<ms hex><random hex>), so a plain listing is chronological.

MCP server

s3-agent-memory serve -n <namespace> starts an MCP server on stdio (official mcp SDK 2.x, MCPServer). The namespace is fixed at startup: no tool takes a namespace, so one user's agent cannot reach another user's memories.

--read-only keeps recall and list usable and refuses writes. Client snippets are in examples/: Claude Code (claude mcp add memory -- s3-agent-memory serve -n users/me), Cursor, and any client that launches a stdio command.

LangGraph store

from s3_agent_memory import MemoryStore
from s3_agent_memory.langgraph_store import S3Store

store = S3Store(MemoryStore.from_url("s3://my-bucket/agents"), index=["text"])
graph = builder.compile(store=store)          # nodes receive `store: BaseStore`
store.put(("users", "alice", "prefs"), "units", {"text": "metric"})
store.search(("users", "alice"), query="units", filter={"text": "metric"})

S3Store implements batch/abatch for LangGraph's GetOp, PutOp, SearchOp and ListNamespacesOp (pip install "s3-agent-memory[langgraph]"). Values are stored in the memory's data; the text of the index fields (or every string in the value) is what gets searched. Filters support equality and $eq $ne $gt $gte $lt $lte. TTLs are in minutes with refresh_on_read, as in LangGraph. Keys that are not safe ids are stored under a SHA-256-derived id. See examples/langgraph_agent.py.

CLI

s3-agent-memory remember TEXT [--kind K] [--tag T]... [--importance F] [--ttl-days N] [--source S] [--id ID]
s3-agent-memory recall QUERY [--kind K] [--tag T]... [--limit N]
s3-agent-memory get ID | forget ID [--reason R] | update ID [--content C] [--importance F] [--tag T]... [--expected-version V]
s3-agent-memory list [--kind K] [--tag T]... [--limit N]
s3-agent-memory consolidate [--kind K] [--threshold F] [--summariser ollama|longest] [--model M] [--dry-run]
s3-agent-memory prune | reindex | stats | namespaces [PREFIX]
s3-agent-memory export [-o FILE] [--include-expired] | import FILE [--overwrite] [--resurrect]
s3-agent-memory serve [--read-only]
# every command: [--url URL] [-n NAMESPACE] [--embeddings none|fastembed|sentence-transformers] [--embedding-model M]

Output is JSON. Exit codes: 0 ok, 1 not found, 2 validation error, 3 conflict or backend error.

Configuration

Limits (Limits(...) on MemoryStore): content 16 KiB, data 32 KiB, metadata 4 KiB, 32 tags, 1 MiB per memory object read, 64 MiB per index or shard read, 100 000 memories per namespace, 2 000-character queries, 100 results, 128 KiB per JSONL line.

Extras: [s3] boto3, [embeddings] numpy (bring your own embedder: any object with name and embed(texts)), [fastembed] numpy + fastembed, [langgraph]. The core needs only the mcp SDK.

Server-side encryption. On AWS, prefer bucket default encryption (SSE-S3 or SSE-KMS with a bucket key) and Block Public Access; S3_AGENT_MEMORY_SSE sets the header per object when the bucket policy requires it. Turn on bucket versioning if you want undo for forget beyond the tombstone; add a lifecycle rule on noncurrent versions so deleted memories really go away.

Local S3

docker compose up -d            # SeaweedFS S3 API on 127.0.0.1:8333, bucket agent-memory
cp .env.example .env && set -a && . ./.env && set +a
python examples/s3_demo.py      # typed memories, 8 concurrent writers, a refused stale update

MinIO works too (docker compose --profile minio up -d with MINIO_IMAGE set): upstream stopped publishing community images in October 2025, so SeaweedFS is the default. Both support the conditional writes this library relies on.

Security

Memory content is untrusted input that ends up in a model's context. Recall returns it as a JSON field with provenance (object key, source, timestamps, version), the MCP envelope says it is data not instructions, and the server instructions tell the model not to follow requests found inside memories. Keys are allow-listed (no .., /, \, %, NUL, reserved names), the filesystem backend refuses symlinks and paths outside its root, every read is size-bounded, vector shards load with allow_pickle=False after a zip-bomb check, tampered objects are refused, and no log line contains memory text or credentials. Threat model and reporting: SECURITY.md.

Testing

python -m venv .venv && . .venv/bin/activate && pip install poetry && poetry install -E s3 -E embeddings -E langgraph
poetry run pytest                                   # 238 tests, coverage ~97%
poetry run ruff check s3_agent_memory tests examples && poetry run mypy s3_agent_memory
S3_AGENT_MEMORY_LIVE_ENDPOINT=http://127.0.0.1:8333 poetry run pytest -k s3-live --no-cov   # real local S3

Every backend test runs on the in-memory, filesystem and S3 (moto mock_aws) backends, and on a real SeaweedFS server in CI. Concurrency tests start threads behind a barrier (remember, update, strict expected_version), inject a writer between read and conditional write to prove the 412 is retried, and check give-up after max_retries. Security tests cover path traversal and key injection, symlinks, tampered index/memory/shard objects, size limits, namespace isolation through every interface, and that logs carry no content or credentials. The consolidation summariser is tested against a fake local ollama HTTP server. Nothing calls AWS or a hosted model.

Part of the series

Contributing

See CONTRIBUTING.md, ARCHITECTURE.md, SECURITY.md and the agent contract in CLAUDE.md.

License

MIT © 2026 Martín Castro Álvarez. See LICENSE.