engram

memory for work that continues after the conversation ends.

website · documentation · 0.8.1 changelog

i built engram to keep the things an agent should be able to return to: decisions, errors, project context, procedures, and the connections between them. it stores memories in sqlite or postgres, searches them through several retrieval signals, and exposes the same store through a CLI, an MCP server, and a web workspace.

it also has a lifecycle. memories can be edited, challenged, superseded, promoted, or forgotten. retrieving something is not proof that it is still true. keeping that distinction visible matters more than making every query return an answer.

how it works, in plain language

engram gives an AI a memory it can return to across conversations, projects, and different tools. the useful part is how it decides what to keep, what to bring back, and whether that information is still relevant.

imagine spending an evening fixing an app. you make a few decisions, discover why something broke, try a fix that fails, and leave one thing unfinished. a new chat would normally need that explained again. with engram, the agent can save those useful pieces and retrieve them when the work continues.

it works roughly like this:

1. save useful information. the agent can store a fact, a decision and its reasoning, a procedure, an error to avoid, or a summary of the session. you can also import notes and documents. it keeps details such as when something was saved and where it came from.

2. find it in several ways. when you ask a question, engram searches by meaning, matching words, and connections between people, projects, and other stored information. asking "why did we change the database?" can find a note about "moving storage to Postgres for concurrent access," even though the wording differs.

3. check which matches actually matter. an optional second model looks more closely at the question and the shortlisted memories. with that reranking enabled, engram ranks them and applies your relevance cutoff. that can mean returning only a few memories, or none. the explanation feature shows which candidates were considered and why they passed or failed.

4. give the relevant context to the AI. the agent receives those memories and uses them while answering or working. the underlying AI model stays the same; it now has useful notes available alongside your current request.

there's also maintenance. a decision from last month might be outdated today. you or the agent can edit a memory, mark it as superseded, challenge it, or forget it. consolidation can combine related material and turn repeated experience into more reusable knowledge. session handoffs can record where work stopped and what should happen next, so resuming involves more than searching for a few vaguely related sentences.

you control the storage and settings. it can keep memories locally, use a shared database, run local models, or use supported hosted providers. different connected agents can work with the same memory store.

the limitation is that it can still save bad information, miss a useful memory, or reject something relevant. retrieving a note doesn't prove the note is true. what engram adds is continuity, and a clearer way to inspect why that continuity worked or failed.

Related MCP server: Enhanced Cognee

what lives here

• retrieval: dense embeddings, full-text search, entity relationships, and associative retrieval, combined with intent-aware ranking and optional rerankers.

• memory: working, episodic, semantic, procedural, and codebase layers, with separate fact, procedure, and narrative types.

• continuity: session diaries, structured handoffs, recent context, decisions, error patterns, and explicit negative knowledge.

• maintenance: deduplication, consolidation, retention, entity graph tools, filesystem drift checks, and status history.

• inspection: a web dashboard, retrieval explanations, query comparisons, activity views, exports, and direct access to the underlying database.

• dormant recall: an optional shadow experiment that looks for a neglected relevant connection without putting it into ordinary answers.

some write and maintenance paths use an LLM for extraction, enrichment, or consolidation. local embedding search can run without an LLM service. choosing an API embedding or LLM backend sends the relevant inputs to that provider.

what's new in 0.8.1

the demo now walks through a fictional project using current setup, recall and retrieval explanations. run engram demo, or engram demo --yes for an unattended walkthrough. its configuration and memories stay separate from yours.

the harmless BGE position_ids loading report is quiet. genuine missing-weight, shape and model-loading problems still show up; model scores are unchanged.

what arrived in 0.8.0

engram init guides storage and local model selection, creates a private new store, and prints connection settings for your agent. --yes supports unattended setup. existing files and client settings stay untouched.

engram doctor checks configuration, storage and model readiness. --full runs real model inference, an isolated local MCP handshake, and a synthetic save/retrieve cycle without an LLM. these checks leave your memories unchanged; the local handshake checks Engram's endpoint, not external agent attachment.

local reranking can now retry a focused excerpt for a confidence-rejected long memory even when another candidate blocks the early retry. excerpt selection avoids repeated unrelated boilerplate beside a complete matching fact while retaining unique context and recognized qualifications. the existing confidence cutoff and one-excerpt-per-memory limit still apply.

reproducible synthetic checks cover this production retrieval path separately from the historical LongMemEval development result below.

benchmarks

100.0% session recall-any@5 (470/470) on LongMemEval-S cleaned, confirmed by a fresh full run on September 20, 2026. every embedding and reranker score was recomputed; no saved inference scores were reused. the run had 0 top-five misses. rank-one recall was 439/470 (93.4%).

this is a development-set measurement: the dataset was also used for tuning. the evaluation skips 30 abstention questions and disables the production confidence cutoff. it measures whether at least one labelled answer session appears in the first five results. generated-answer accuracy is not measured. the measured retrieval candidate was published in commit c3b8cea, before the configuration, explanation and website changes in 0.7.0. the linked evidence retains that run's exact source hashes; this is not a second full benchmark of the release package.

method and limitations · verified result and provenance · question-level session rankings

engram and other systems

these are reported figures with their actual metrics. external systems were not rerun here, and their metrics, question sets and evaluation setups differ. the table is a reference comparison, not a shared ranking.

the previous table incorrectly put answer accuracy under an R@5 heading and converted other systems' percentages into misses out of 470. those comparisons have been removed. LongMemEval's retrieval methodology excludes the 30 abstention questions; QA evaluation measures a different task.

by question type — fresh run

run the same configuration locally on Apple Silicon:

python benchmarks/longmemeval/run_engram.py data/longmemeval_s_cleaned.json \
  --rerank --embedding-backend mlx --output fresh-results.jsonl

use a new output path for each run. the published evidence records dataset and source hashes, model revisions, package versions, settings and all question IDs.

get it running

python 3.11 or newer is required:

pip install --upgrade engram-memory-system
engram --version
engram init

engram 0.8.0 includes guided setup and doctor checks.

init guides storage, local model selection and new config/database paths. it creates a private config and initialized store, then prints an MCP connection snippet for your agent. it does not edit client settings or download models. existing config, database and index files are refused.

for unattended setup with explicit new paths:

engram --config /absolute/new/config.yaml init --yes --preset portable \
  --db-path /absolute/new/memory.db
engram --config /absolute/new/config.yaml doctor --full

local uses BGE embeddings and reranking with automatic local runtime selection. portable selects sentence_transformers; light also uses the smaller MiniLM reranker. the runtime chooses its device. without custom paths, setup uses ~/.config/engram/config.yaml and ~/.local/share/engram/memory.db.

doctor --full checks configured models, an isolated local MCP endpoint, and a synthetic save/retrieve cycle without an LLM. it may download weights or contact a configured model provider with synthetic text. user memories and client settings remain unchanged; SQLite reads can use normal WAL/SHM bookkeeping. the local MCP check establishes endpoint behavior, not attachment by another agent.

plain doctor checks configuration, existing storage and package/cache metadata. incomplete means optional checks were skipped or readiness remains unverified; failed checks exit 1 and invalid configuration exits 2. use the exact command printed by init when you chose a custom config path.

to install the code in this checkout instead:

git clone https://github.com/raya-ac/engram.git
cd engram
python3 -m venv .venv
. .venv/bin/activate
pip install -e ".[dev]"
engram init

the default embedding model is BAAI/bge-small-en-v1.5; the default reranker is BAAI/bge-reranker-base. models use cached weights after first download.

inspect the generated config before storing your own material. it controls the database, models, LLM backend, web binding and optional experiments. extraction and enrichment use the configured LLM backend, which defaults to the Claude CLI. the setup and doctor checks above do not require an LLM.

engram --config /absolute/path/to/config.yaml config check
engram --config /absolute/path/to/config.yaml config show --json
engram config schema --json
engram --config /absolute/path/to/config.yaml remember "the release requires a restore drill before activation"
engram --config /absolute/path/to/config.yaml search "release preparation" -k 5 --json
engram --config /absolute/path/to/config.yaml search "release preparation" --rerank --explain --json

--config comes before the command. without it, engram checks the current working directory, the source checkout, then ~/.config/engram/config.yaml. every field supports an ENGRAM_* environment override, such as ENGRAM_RETRIEVAL_TOP_K=5. environment values take precedence over the file; see configuration and the example. engram config show --defaults inspects package defaults without reading local files or environment overrides. setup reports inherited override names without printing their values; make required variables available to the agent process. existing configuration files are never rewritten.

for a new Postgres store, set ENGRAM_POSTGRES_DSN in the environment and run engram --config /absolute/new/postgres.yaml init --storage postgres --yes. the database must already exist with an empty current schema. setup leaves the DSN in the environment and never clears existing tables.

storage and models

sqlite is the default. it uses FTS5 and stores its database at ~/.local/share/engram/memory.db. postgres uses the same memory model with its own full-text search and connection adapter.

storage_backend: sqlite
db_path: ~/.local/share/engram/memory.db
embedding_backend: sentence_transformers
embedding_model: BAAI/bge-small-en-v1.5
embedding_dim: 384
cross_encoder_model: BAAI/bge-reranker-base

for postgres, set storage_backend: postgres and provide ENGRAM_POSTGRES_DSN or a private postgres_dsn config value. keep credentials out of commits. engram migrate-postgres --help describes the existing sqlite migration tool. back up the source first and check its verification output before switching.

local embeddings and cross-encoder reranking are included in the base install. cross-encoder/ms-marco-MiniLM-L-6-v2 remains an optional local reranker. existing config files keep their selected model; changing the package default does not rewrite them.

optional extras provide the API clients:

pip install -e ".[voyage]"
pip install -e ".[openai]"
pip install -e ".[gemini]"
pip install -e ".[api]"

model selection determines the embedding backend for recognized API models. configure the corresponding provider key in the process environment. changing embedding models requires compatible dimensions and re-embedding existing data; use engram reembed --dry-run before engram reembed.

use it from an agent

building memory into your own app? the integration guide links usable source for an Open WebUI filter, a Minecraft Paper plugin, a game-server bridge, and a task app assistant. the website tutorials explain configuration, adaptation and verification.

there is also a Discord channel bot, an NPC memory adapter, and a shared sync/async checkpoint client. other Paper plugins can call the registered EngramMemoryService directly. the what you can build guide walks through returning-player NPC dialogue, quest journals, build companions, town lore, community handoffs and chat assistants, with the source to start from.

setup guides now cover Codex, Claude Code, Claude Desktop, Cursor, VS Code / Copilot, Windsurf / Cascade, Cline, OpenCode, and Gemini CLI. the client hub explains shared stores, actual connection checks and custom integrations. each client has its own config format; the JSON below is an example for clients using mcpServers.

start a stdio MCP server:

engram --config /absolute/path/to/config.yaml serve --mcp

an MCP client can launch it with an explicit executable and config:

{
  "mcpServers": {
    "engram": {
      "command": "/absolute/path/to/engram/.venv/bin/engram",
      "args": ["--config", "/absolute/path/to/config.yaml", "serve", "--mcp"]
    }
  }
}

use recall_recent for chronology and recall_hints for lightweight recognition. use recall for a concrete semantic query. remember, remember_decision, and remember_error store context; session_handoff and resume_context support resuming work. tools also cover entities, timelines, memory status, codebase scanning, drift, and consolidation. the server's tools/list is the current interface; mcp_server.py defines it.

recall supports facts_only, facts_plus_rules, and full_context profiles. ordinary search records returned results as accesses. that is a retrieval signal, not explicit evidence that someone used the result. use recall_explain to inspect retrieval without recording those accesses, or config_show to inspect redacted effective settings. rejected candidates remain visible even when no memory passes the confidence gate; forgotten, inactive and profile-filtered memory content stays hidden.

native application integration

engram --config /absolute/config.yaml api runs a persistent local JSONL process. it exposes project context, semantic search, session checkpoints and structured evidence through a supported service boundary. a harness can launch Engram in its own Python environment without importing the store or speaking MCP.

Kiln's native experience and Mythic's check/planning lifecycle belong to those projects. Engram supplies reusable memory and evidence APIs. evidence records carry caller provenance, timestamps, expiry and source references; they are not independently verified merely because they were stored. referenced forgotten or inactive memories cannot remain eligible supporting evidence. no observation can instruct Engram to execute a command or fetch a URL.

the native API contract distinguishes nonreinforcing scoped context from ordinary semantic search, which records accesses. it also describes immutable evidence, explicit unknown/stale states and supported operations.

optional Codex compatibility

the optional separate adapter exposes codex_context, codex_checkpoint, codex_diagnostics, and a scoped codex_evidence reader. it reads explicitly scoped active memories, saves a deliberate task handoff, and reports its own process and persisted index coverage. it doesn't alter core retrieval, collect transcripts, install hooks, or reinforce memories when context is read.

engram --config /absolute/engram/config.yaml codex setup --project /absolute/project

that prints a supported codex mcp add command for review; it does not register anything automatically. each adapter process is bound to one canonical project directory. unscoped legacy notes and sibling projects stay outside its context. checkpoints are separate from memory records and can be replaced or cleared. the full Engram MCP server can remain connected alongside it for broader recall.

adapter setup, task-start/resume workflow and limits includes tested registration syntax and explains what diagnostics can establish. a fresh adapter's PID does not prove another running client has reloaded code.

how retrieval works

query → intent and query features
      → dense + full-text + entity graph + associative candidates
      → reciprocal rank fusion
      → date, importance, recency and access boosts
      → optional cross-encoder with confidence gate and prior coverage
      → optional trained deep reranker
      → noise only when cross-encoder is off → ordinary results

original query → optional independent dormant search → separate shadow log

the ANN index accelerates dense retrieval when available; a brute-force embedding path remains available. the CLI makes cross-encoder reranking opt-in with --rerank; MCP recall uses it. the deep reranker runs when a trained model is available. debug output explains the ordinary ranking stages.

retrieval.rerank_passage_fallback is enabled by default for local rerankers. when every full-document score, after sigmoid, is below rerank_passage_floor (default 0.001), it can retry one excerpt of up to 160 words from each long document with matching query terms. production search also retries eligible long memories rejected by the final confidence gate, even when another candidate scores highly. each document still gets at most one excerpt. it keeps the larger full-document or excerpt logit and uses the same semantic query for both calls. short documents, documents without a lexical match, and hosted rerankers keep their original scores. set rerank_passage_fallback: false to disable the retry.

excerpt matching includes conservative regular English singular/plural forms; each distinct original query term counts once per sentence. the excerpt can omit repeated unrelated neighboring boilerplate when the matching sentence covers every query term; unique context and recognized qualifications are retained. the early activation floor was selected during development on LongMemEval. results on that dataset are development measurements, not held-out accuracy, and the floor is not a calibrated probability.

the retry adds model work and can leave useful context outside the excerpt. the independent min_confidence gate (default 0.6) applies after scoring adjustments and can reject a result whose benchmark rank improved. retrieval internals describe selection, date handling and score traces.

ranking scores depend on the stage and model. a cosine score, a cross-encoder score, and a fused ranking score are not interchangeable confidence estimates. local cross-encoder logits pass through a sigmoid once; hosted rerankers keep their normalized score scale. resolved temporal evidence is applied in logit space. retrieval.rerank_fusion_alpha optionally blends the resulting score with the pre-rerank reciprocal rank, with a value between 0 and 1; its default is 0. reranked scores stay between 0 and 1, without additional lexical bonuses or random noise. searches with reranking off retain the small random noise term.

retrieval.preserve_prior_candidate is enabled by default. after the confidence gate, a request for at least two results keeps the best eligible hybrid candidate in the requested result count. if it is missing, the coverage step moves it into the last requested position while retaining the rerank winner. this changes selection order without changing score values, including when rerank_fusion_alpha is 0. consumers should keep the returned order rather than sort it again by score. set preserve_prior_candidate: false to disable coverage.

changing the fusion weight, minimum confidence, coverage setting, passage fallback flag or activation floor uses a separate rerank cache entry. the production search and benchmark share date parsing and relative time helpers. search preserves the user's spelling; configured query expansion still applies.

dormant recall

sometimes an old project stopped because a capability did not exist yet. later, a new capability appears, but ordinary retrieval keeps returning the recent, often-used material. dormant recall asks whether a neglected memory is relevant to the task now.

this is an experiment. it is off by default. enabling shadow mode evaluates at most one candidate per hybrid search, or none, and writes the result separately. it does not add suggestions to recall output, hints, context, or normal answers. there is no automatic visible mode in this version.

dormant_recall:
  mode: shadow                 # use "off" to disable; quote it in YAML
  candidate_limit: 50
  dormancy_days: 30
  min_relevance: 0.75
  max_bonus: 0.05
  rerank_candidates: 12
  min_rerank_score: 0.6
  cooldown_days: 7
  feedback_cooldown_days: 30
  log_max_events: 1000
  log_retention_days: 30

the candidate search reads current active, dormant embeddings directly from the database, independently of ordinary results and the ANN cache. a stale index cannot hide a newer memory from this path. candidates must pass the cosine floor and a separate query/content relevance check. a bounded bonus from dormancy and existing importance can reorder close relevant matches; age cannot bring an irrelevant item through the gate. literal word overlap is supporting evidence, not a requirement, so differently worded connections can still qualify. the explanation reports the evidence without inventing a story about why an old problem is now solved.

only active, non-forgotten memories qualify, with the same memory-type profile as the search. forgotten, archived, deleted, superseded, merged, or otherwise inactive memories are excluded. the current forgotten flag covers both lifecycle archival and explicit forgetting; this experiment does not try to separate them.

review it explicitly:

engram --config config.yaml dormant review --limit 20
engram --config config.yaml dormant inspect EVENT_ID
engram --config config.yaml dormant feedback EVENT_ID useful

review lists metadata. inspect fetches the eligible candidate and records a separate exposure timestamp. feedback is useful, irrelevant, or dismissed; useful means the connection was actually used. MCP provides dormant_review, dormant_inspect, and dormant_feedback for the same workflow.

computing, inspecting, or giving feedback on a candidate never increments its ordinary access count, updates its ordinary last-accessed time, or changes its importance. silence does not mean useful. cooldown and explicit-use state survive log rotation. the bounded log stores IDs, times, numeric signals, and feedback, without copying query text, matching words, or memory content.

set ENGRAM_DORMANT_RECALL_MODE=off to override the file at process start. existing MCP processes do not hot-reload these edits. the feature creates only additive experiment tables and contains its failures so normal recall can finish. see the dormant recall guide for ranking, persistence, retention, and feedback details.

what is verified: isolated SQLite and PostgreSQL tests cover eligibility, independent candidates, bounded ranking effects, cooldown, feedback, persistence, concurrency, failure isolation, and absence of reinforcement. a separate CLI/MCP smoke test uses synthetic memories and the cached local embedding model.

what remains unproven: usefulness in real work and threshold calibration. in the initial cosine-only probe, a sentence with no literal query overlap scored about 0.734. the 0.75 default abstained; a lower diagnostic threshold admitted it. the added relevance check now rejects that thin sentence. the current interface smoke test uses a more informative prototype note, which still falls below 0.75 and is admitted only in its separately labelled 0.70 isolated pilot. that is a concrete precision/recall tradeoff, not evidence of broad usefulness.

the repair also recovered a real Junkstep release note with one previous access and about 37 days of dormancy. it was missing from the old ANN index but ranked first against current database vectors at 0.817. in an isolated snapshot it added installer-packaging and release-verification context missing from ordinary results, passed the relevance check, and remained unreinforced. this is a targeted evaluator assessment; no positive user feedback was fabricated. exact scanning trades additional database work for current coverage, so larger stores need measurement before treating this as a scaling solution.

keep the store understandable

engram ingest /path/to/notes/
engram entity "project name" --graph
engram drift --json
engram patterns --dry-run
engram index status
engram export memories.json --include-embeddings
engram import memories.json --skip-duplicates
engram consolidate

consolidation can cluster, summarize, promote, and soft-archive memories. drift checks compare stored filesystem references with current files; drift --fix changes memory state. inspect these operations before using them on a valued store. they are maintenance actions, not substitutes for validating a claim.

the portable export includes active memories. it is not a lossless backup of forgotten records, all history, or the dormant experiment tables. use native sqlite or postgres backup tooling when you need the full store, and verify a restore into an isolated database. the legacy sqlite-to-postgres migration does not yet transfer dormant telemetry or its cooldown state.

inspect it in the browser

engram --config config.yaml serve --web

the workspace binds to 127.0.0.1:8420 by default. its redesigned memory list, navigation and inspector share a restrained archive layout. all eighteen original views remain available, alongside dormant review. mobile Menu and Activity drawers retain navigation and inspection; / focuses search, and view URLs can be bookmarked. memory editing loads the full record rather than the list excerpt.

web workspace controls cover search, hints, filters, continuity, graph and maintenance workflows. web.auth_token configures bearer authentication. keep the database and config private, and check authentication and network exposure before making the service reachable outside the local machine. serve --mcp-sse is also available for HTTP MCP use.

development and checks

python -m pytest tests/ -q

the ordinary suite expects cached local embedding weights; the shared fixtures set Hugging Face offline mode. dormant contract tests use temporary sqlite files. set ENGRAM_TEST_POSTGRES_DSN to a disposable test cluster to also run their postgres cases; each creates and drops its own randomly named schema.

python -m pytest tests/test_dormant.py -q
python tests/dormant_smoke.py --work-dir /path/to/new/disposable/directory

the smoke test starts fresh CLI and stdio MCP processes against synthetic sqlite stores. it compares ordinary result IDs with the experiment off and on, exercises review/inspection/feedback, checks cooldown, and reads back access and importance fields. it writes a JSON report in the supplied directory. it does not open a production config or restart an existing service.

config, store, retrieval, dormant recall, lifecycle, and the MCP server are the main entry points; the native service is the application boundary and the optional Codex adapter is a separate integration layer. further guides and reference material live in docs.

license

Engram Public Use License 1.0.