Lethe Akribeia

An experimental long-term memory runtime for AI agents

[ English | 日本語 ]

“Forgetting is not deletion. It is loss of resolution.”

A memory system that treats forgetting as progressive resolution loss rather than deletion. Deterministic memory compilation · temporal reasoning · evidence provenance · MCP native (Formerly: Artificial Memory / lethe-akribeia)

Measurement status — read this before quoting any number

1. LoCoMo, official token F1 (pinned upstream scorer, 1,986 questions). The originally published pipeline measures 37.41% (categories 1–4: 48.24%). Two measured changes move it: emitting the benchmark's canonical refusal form (not mentioned) gives 49.91% — a format fix, not a capability gain — and the answer-provenance refusal gate plus turning the deterministic committer off gives 54.94%. The last figure is a composition of paired deltas (53.77% with the gate, +1.16pt from removing the committer), not a single stored run. The v0.3.0 artifact re-scored on the same protocol is 51.89%. Source: design_docs/v4_world_class/V4_PROGRAM.md §2, §4.9, §4.17; artifacts in benchmark_results/official_locomo_score_v4r7_aligned.json, ..._v4r7_refusal_form.json, ..._v030_rules_commit.json.

2. The binary 65.26% is not an official F1 and is not comparable to one. It comes from an in-house deterministic matcher, and for LoCoMo category 5 the old harness used adversarial_answer as gold. That means 126 category-5 answers were scored "correct" for matching the distractor; the official scorer gives that category 0.00%. The 65.26% figure is categories 1–4 only. Source: V4_PROGRAM §3.1.

3. LongMemEval numbers (98.20% oracle recall, 83.40% end-to-end) are v0.3.0 artifacts. They come from one 500-question run with a local 7B reader and an in-house scorer. Published LongMemEval results in the 95%+ range use different answerers and LLM judges; no head-to-head under a shared harness exists here.

4. BEAM 20/20 at 500K is a needle-in-haystack probe, not a BEAM leaderboard result. Published BEAM numbers use LLM judges. This is an in-house binary matcher over 20 questions at 500K tokens.

5. The shipped runtime's default configuration scored 11.5% on the controlled 200-question arena. An IDF-weighted lexical recall channel lifts it to 53.0% (Simple RAG 51.0%, Mem0-style 42.5%, Zep-style 35.0%, HippoRAG-style 26.0%) at roughly 3× the read tokens. That channel is now on by default (--no-lexical or LETHE_LEXICAL_RECALL=0 disables it); it was not reachable from the CLI/MCP path when the arena was measured. Source: V4_PROGRAM §4.12–§4.14.

6. A holdout leak is recorded, not hidden. An inverted --scope filter caused one run of the provenance gate to measure only the 499 held-out questions. That gate's holdout is therefore no longer clean and cannot be cited as validation. The separate 553-question holdout has not been spent. Source: V4_PROGRAM §4.17.

7. The tag v4-r8.1-perfect-score is a development-set artifact, not a result. It records 250/250 on LongMemEval dev-250, produced with 56 deterministic overrides (22.4% of answers), and it is not part of the V4 measurement log. Dev saturation is not held-out accuracy and must not be cited.

If you only remember one thing: the honest headline is 54.94% official LoCoMo F1 with two non-default settings, and 53.0% vs 51.0% for Simple RAG on one 200-question synthetic arena — not "Apex", not "state of the art", not "100%".

Related MCP server: cognitive-harness-mcp

What this repository is

Two Python packages, a CLI, an MCP server, and the benchmark harnesses that produced the numbers above.

There is no training code and no model weights in this repository. All ingestion is deterministic; models are used at answer time and, in some research paths, for retrieval assistance.

How to read our numbers: three layers

High retrieval recall does not imply high end-to-end accuracy, so we report them separately and never multiply them into one headline.

  Layer 1 (Oracle Recall)  ── did the pipeline retrieve/compile the gold evidence turns?
  Layer 2 (Reader on Hits) ── given that evidence, did the Reader answer correctly?
  Layer 3 (End-to-End)     ── final pipeline accuracy (query -> retrieval -> reader -> answer)

v0.3.0 release artifacts (as published at v0.3.0)

These are the numbers that were published with v0.3.0. Read the warning above first: the binary column is an in-house matcher, and the official F1 for the same LoCoMo artifact is 51.89%, not 65.26%.

On the temporal slice, the autonomous committer's own claims were 70.64% precise (77/109) on the frozen run and 68.89% (124/180) on a later offline sweep. It was subsequently A/B'd and made opt-in, because the reader scored +10.31pt higher on the same 312 claims.

Current status on the official protocol

Pinned upstream LoCoMo scorer (task_eval/evaluation.py, metric=f1), 1,986 questions:

Two caveats that matter for reproducibility: the 54.94% figure is a composition of paired deltas rather than one stored run, and the refusal-form gain is a format fix. Neither is a new reasoning capability.

Controlled arena (200 planted-fact questions)

All players share one frozen answer LLM, one frozen extraction LLM, one embedding model, a 2,000-token budget and one scorer:

The arena is synthetic, plants facts, and was run once per player at n=200. The 60.0% conflict-surfacing row is a player-side measurement; the same mechanism in the shipped product path moved 40.0% → 42.0% on a paired 50-question run with one contradiction conversion out of five. It is not yet a default. Source: V4_PROGRAM §4.12–§4.21, §4.24–§4.30.

What is measured as not working

Keeping this list is the point of the project. Each row was implemented or priced and then rejected on measurement.

Core mechanics

Forgetting = loss of resolution, not deletion

The runtime defines six resolution levels; the default pipeline writes the lower ones and consolidates upward as memory ages and is re-used.

Decay is triggered by turn age, access recency/frequency, and a Utility/Token budget score. Retrieval starts at low resolution and expands only when needed.

Zero-LLM ingestion and deterministic co-processors

• Write LLM calls = 0. Ingestion and indexing are regex, syntactic parsing and a token inverted index.

• Temporal co-processor: calendar arithmetic, relative expressions resolved against message timestamps.

• Aggregation co-processor: counting, lists, frequencies.

• Answer committer: a strict answer contract. Measured to cost points on LoCoMo, so it is opt-in, not default.

• Evidence binding: a clause-local contract that refuses to attach a value to an event it cannot see. It deliberately does not resolve paraphrase — see limitations.

Context as an intermediate representation (MSC)

Rather than dumping raw text into the prompt, memories compile into a structured Memory State Context:

[TEMPORAL_ANCHOR] Reference Date: 2023-05-14
[VERIFIED_STATE] User lives in Kyoto (as of 2022-10)
[DETERMINISTIC_TIMELINE]
 - 2023-01-10: Visited Paris
 - 2023-03-04: Switched job to Researcher
[EVIDENCE_CHAIN (Provenance)]
 - [conv_3:turn_12] User: "I am taking the exam next month..."

Vocabulary governance

Every knowledge table (person rosters, nickname maps, synonym expansions, query patterns, duration anchors) lives in src/artificial_memory/ontology/ as versioned YAML with a _meta.origin field separating world knowledge (generic) from corpus-derived tuning (benchmark_tuned). A CI leak guard (tests/unit/hygiene/) scans runtime code for benchmark fingerprints; comments and docstrings are exempt, executable code is not, and the shrinking allowlist is itself tested. This is the mechanism that makes "no benchmark vocabulary in engine logic" a checked property rather than a claim.

What you can do today

MCP server

python -m artificial_memory.mcp

Tools: memory_remember, memory_recall, memory_expand, memory_trace, memory_timeline, memory_explain, memory_inspect.

CLI

git clone https://github.com/Yato-Works/Lethe-Akribeia.git
cd Lethe-Akribeia
pip install -e ".[vector,llm]"

lethe start "Project/Akribeia"
lethe user "We decided to target 9/30 for the initial release."
lethe assistant "Understood. The release deadline is set to September 30."
lethe end
lethe recall "When is the release scheduled?"     # works in a new process
lethe ask "When is the release scheduled?"        # evidence + compiled context
lethe resume "Project/Akribeia"                   # continue the same topic
lethe timeline

ask returns what the agent would send to its model: the matching memories, their provenance, and the compiled MSC context. It does not call an LLM, so it is not a generated answer. recall/timeline no longer require an active conversation; they fall back to the most recent topic unless --topic is given.

Known limitations

These are current, measured, and mostly queued for the next milestones.

• Memory quality on real dialogue is modest. 54.94% official F1, with the single-hop and adversarial categories carrying most of the loss.

• Recall is tuned on a synthetic arena, not on your token bill. The lexical channel is on by default because it is the measured-better configuration (11.5% → 53.0%), but it reads roughly 3× the tokens. Disable it with --no-lexical or LETHE_LEXICAL_RECALL=0 when budget matters more. Conflict surfacing remains off; its product-path transfer is still partial.

• No paraphrase resolution. Evidence binding is clause-local: if the duration and the event sit in different comma-clauses, or the question paraphrases the event ("creating art" vs "painting"), the engine declines rather than guesses. This is covered by two explicit tests (one pass, one strict xfail).

• Cue detection does not model negation. "Nothing has been chosen yet" still registers as a decision cue, in Japanese and English alike. The markers are deterministic word matches, not a parser.

• Test coverage is 54%. Many research modules (artificial_memory_gen, compiler/eval) are exercised only by benchmark scripts, not by CI.

• Type checking is not clean. mypy reports ~1.4k findings, dominated by untyped third-party decorators and missing stubs. Ruff is clean and blocking.

• Two stacks, one repo. artificial_memory and artificial_memory_gen are not merged; RuntimeFacade carries 95 methods (89 public). All 10 compiler/stages classes are wired into the default pipeline — measured, not dead — so the remaining work is the merge itself, not deleting stage modules. Merging is planned work, not done work.

• API defaults are loopback-only, but the service is not a production deployment target. CORS defaults to localhost origins; LETHE_ALLOW_ORIGINS="*" is accepted only with credentials disabled. python -m artificial_memory.api.server binds 127.0.0.1, and LETHE_ENABLE_AUTH=1 now actually enforces a session or API key on every non-public route (before this change the flag protected nothing). There is still no TLS, rate limiting, per-user isolation, or WebSocket authentication.

• Benchmark scripts need local models (Ollama, a reader model, optionally embeddings). CI runs unit tests only; it does not reproduce the tables.

Smoke check (one command)

Before and after any pipeline change, run the pinned smoke set. It sends real LoCoMo questions end to end through the same frozen path as the full benchmark and shows you exactly what the reader was given:

# 18 pinned cases, real 7B reader (needs a local Ollama)
python scripts/benchmarks/locomo_smoke.py

# a few cases only
python scripts/benchmarks/locomo_smoke.py -n 4

# no reader at all: 0 LLM calls, still verifies retrieval + context compilation
# run and shows how much evidence each case would receive
python scripts/benchmarks/locomo_smoke.py --no-llm

Output lands in benchmark_results/smoke/<stamp>/: report.md (per-case table, category rollup, failures in detail, the exact reader input), results.json, reader_inputs/, and run.log. The case set is pinned in benchmark_config/smoke_manifest.json and excludes conv-48, the strict clean holdout. These numbers are a plumbing signal, not a benchmark score; publishable numbers stay in benchmark_results/RESULTS_REGISTRY.md.

Comparing two runs

--compare diffs two finished run folders: which cases got fixed, which broke, the per-case token delta and the mean context change. It reads the artifacts, so it works on any two runs of the same case set, including one made before a change you are evaluating.

python scripts/benchmarks/locomo_smoke.py                                  # baseline
python scripts/benchmarks/locomo_smoke.py --rerank ollama                  # same cases
python scripts/benchmarks/locomo_smoke.py --compare \
    benchmark_results/smoke/<baseline> benchmark_results/smoke/<second-run>

Re-ranking (--rerank, off by default)

--rerank re-orders the retrieved candidate pool with a learned relevance model before the budgeted compiler selects evidence:

Both fail before the first question if their model is absent. A batch whose reply cannot be parsed keeps its retrieval order and is counted in the manifest's rerank_failures, so a run cannot silently claim it re-ranked anything.

Measured 2026-10-28 on the 18 pinned dev cases (ollama backend): 10 -> 9 correct, 1 fixed / 2 broken, mean context 1410 -> 1258 tokens. Both arms were re-run and reproduced identically (temperature 0, seed 42), so the -1 is a real effect on these 18 cases, not sampling noise — but 18 cases cannot settle the method, and the smaller context is the only structural effect. One of the two breaks is the attribution pair itself: lc_conv-43_178 ("which team did Tim sign with", gold: not mentioned) returned Minnesota Wolves, i.e. John's answer, where the un-reranked run refused. The cross-encoder was never measured here (no PyPI access on the development machine), so the stronger instrument is untested, not refuted.

Reproduction

# 1. Unit tests (680 collected; 676 pass, 2 xfail, 2 xpass)
pytest tests/unit/

# 2. Coverage (measured 54% on 2026-10-26)
pytest tests/unit/ --cov=artificial_memory --cov=artificial_memory_gen

# 3. Lint
ruff check src/ tests/

# 4. Verify the 21 canonical v0.3.0 metrics and their artifacts
python scripts/benchmarks/verify_registry.py

# 5. Re-score a stored LoCoMo run with the pinned upstream scorer
python scripts/benchmarks/score_locomo_run_json.py \
  --results benchmark_results/locomo1540/locomo_1540_improved2.json

# 6. Three-layer decomposition
python scripts/benchmarks/three_layer_report.py

Frozen context caches (benchmark_results/frozen_ctx/) and verifier verdicts (benchmark_results/support_verifier/) are generated inputs and are gitignored; regenerate them with the --freeze-contexts / --out commands named in V4_PROGRAM before running the attribution scripts. The two arena runs cited in V4_PROGRAM §4.21 are committed; 13 other timestamped local runs are not.

The authoritative measurement log, including the negative results, is design_docs/v4_world_class/V4_PROGRAM.md. The 21 frozen v0.3.0 metrics are in benchmark_results/RESULTS_REGISTRY.md.

Status and roadmap

Current: v0.3.3, experimental. The V3 three-layer diagnostic standard and the V4 measurement correction are both done. This branch also closes the product-wiring gap that the arena exposed:

• the measured recall configuration is now the default (--lexical/--no-lexical, LETHE_LEXICAL_RECALL);

• lethe ask and lethe resume exist, and recall/timeline no longer require an active conversation;

• decision/reason/timeline cues are bilingual (Japanese and English);

• the embedding model loads lazily instead of on every CLI invocation;

• exact cross-resolution duplicates (same memory id, or the same body after whitespace normalization) are removed from the context. Measured on a two-turn topic: 20 recall rows across four levels contained 15 repeated ids and 1 repeated body; the compiled context is now 4 parts;

• the API is loopback-only by default and LETHE_ENABLE_AUTH=1 now actually enforces a session/API key (before this change the flag protected nothing).

Still queued:

1. Merge the research and product packages; shrink the 95-method facade. (Measured note: all 10 compiler/stages classes are wired into the default pipeline; no dead stage module was found.)

2. Merge paraphrase-level near-duplicates. The dedupe above only removes exact ids and exact normalized text; it does not merge a fact re-expressed at RAW/EPISODE/SEMANTIC.

3. Deployment hardening for the API (TLS, rate limits, per-user isolation, WebSocket authentication).

No accuracy promise is attached to any of these. Each will be accepted or rejected on the paired measurement described in V4_PROGRAM, not on a demo.

Why release now

I wanted to find out how far a structured, local memory system could go when hardware and models were small: an RTX 3050 (8 GB VRAM), a 7B q4 reader, nothing else. This is a working, auditable checkpoint with measurable strengths and measurable weaknesses. The re-scoring in V4 made the weaknesses visible; that is why the numbers here are lower than the ones in the v0.3.0 release notes.

I still want to find out how far this architecture can go.

Citation

@software{lethe_akribeia_2026,
  title = {Lethe Akribeia: An Experimental Cognitive Long-Term Memory System for AI},
  author = {Yato-Works},
  year = {2026},
  version = {0.3.3},
  url = {https://github.com/Yato-Works/Lethe-Akribeia}
}

License

MIT License. See LICENSE for details.