memory-autopsy

ESLint for LLM agent memory — lint stored facts for duplicates, contradictions and staleness before your agent acts on them.

$ memory-autopsy report examples/memory.md examples/facts.json

# memory-autopsy report

Engine: `heuristic` | Facts scanned: 15 | Findings: 7

| Class | Count | Severity |
|---|---|---|
| `contradicted_explicit` | 2 | error |
| `contradicted_implicit` | 3 | error |
| `duplicate` | 1 | info |
| `unsourced` | 1 | info |

## Findings

- **contradicted_implicit** — entity `billing-store` `examples/memory.md`:5: slot 'anchor:uses' value conflict ('postgresql' -> 'mysql')
  - flagged: "billing-store: uses PostgreSQL 14 for the billing store" (observed 2026-01-05)
  - survivor: "billing-store: uses MySQL 8 for the billing store" (observed 2026-06-02)
- **contradicted_explicit** — entity `Globex` `examples/memory.md`:15: newer fact has transition trigger 'left' on shared anchor 'northwind'
  ...

## Fix plan (dry-run — pass --apply --yes-i-know to execute)

- supersede examples/memory.md:15 — "works at Northwind" (valid_to 2026-05-01)
    new: Globex: employer = "Northwind" ; valid_to=2026-05-01 -- 2026-01-01 (source: profile)
- delete examples/memory.md:11 — "cache-tier: uses redis 7" ([duplicate] identical to 'cache-tier: uses redis 7')

The problem it was built for

A production agent kept acting on stale provider/cron facts its own memory contradicted — an old "uses provider X" claim sat next to a newer "migrated off provider X" note, and the agent kept recommending provider X with full confidence. Nothing in the toolchain noticed: version control tracks the file, not the claims inside it; RAG retrieves both facts and lets the model arbitrate; and paraphrased contradictions share no keywords, so grep cannot see them either.

Agent memory is source code for beliefs. It deserves a linter.

Related MCP server: datapact

What v0.4.0 adds

• Alias-aware entity resolution. Entity identity is no longer exact-match: declared (aka …) aliases and conservative normalized forms (case, punctuation, Ltd/Inc/Corp/Company suffixes) fold into one entity at ingest, so acme and Acme Corp are linted — and time-travelled — as one. Two names that merely look alike are never silently merged: they come back as an unresolved_alias finding for a human to settle. See Alias resolution below.

• Verify-and-confirm contradiction layer. A candidate pair that the dated slot rule cannot sequence (equal dates, one side undated) is now re-checked from structure alone: when both sides are canonical slot = "value" records, slot-equality decides and the conflict is confirmed; when only prose evidence exists, the pair is hedged as likely_conflict with its reason string and a composite confidence (cosine + structured agreement) — never asserted, never auto-fixed. See The grey zone below.

• undo — --apply is reversible. Every rewrite snapshots the file to <name>.bak-<ts> first and prints the path; memory-autopsy undo <file> restores the newest snapshot byte-for-byte. See Undo below.

• 10 grey-zone eval scenarios (58 total) written and labelled before the engines ran, including the two v0.1.0 paraphrase false positives. All declared targets pass under --strict.

What v0.3.0 adds

• Time travel (as-of) — memory-autopsy as-of 2026-03-01 store.jsonl answers what was considered true on that date: valid_from/valid_to windows are honoured, observed_at is the fallback, records with no temporal anchor are reported as unanchored instead of guessed. See Time travel below.

• In-loop write gate (admit) — the SDK call an agent makes before persisting a fact. Returns accepted / requires_review / rejected with the stable ids it disagrees with and the supersede edit that would resolve the conflict. See Write gate below.

• MCP server — memory-autopsy-mcp speaks JSON-RPC over stdio, exposing memory_lint and memory_admit to any MCP client (Claude Desktop, Cursor, Hermes …). Read-only: it never writes a store, never calls the network. See Agent self-audit below.

• Canonical timeline example — examples/timeline.jsonl shows the supersession chain: the same billing store is PostgreSQL 14 until 2026-04-01, MySQL 8 after.

What v0.2.0 adds

• Canonical fact model — a JSONL store, one fact per line, with a stable content-hash id, slot/value, valid_from/valid_to, source_class, fidelity and status_verb.

• Fidelity bands — every record is tagged FULL (an explicit slot = "value" parse), PARTIAL (entity written, claim read as prose) or RESIDUAL (entity inferred). The tag travels into every finding, so a fuzzy parse never presents itself as a proven one.

• Slot-conflict detection (layer 2b) — claims are decomposed into the slots they can prove, and two claims in one slot with different values conflict. Equal heads are refinement, not contradiction.

• Provenance-capped verdicts — reliability is wording weight bounded by source class and fidelity; an unclassified claim can never outrank a primary-source fact, whatever it says and whenever it claims to be from. See docs/threat-model.md.

• Supersede, not delete — a contradicted claim is stamped valid_to and kept as history. Only redundant duplicates are removed.

Findings classes

How a scan works

flowchart TD
    A[Load memory files: markdown, JSON or JSONL] --> B[Parse: explicit slot=value, section blocks, heading blocks, bullets, JSON keys]
    B --> C[Fidelity band + alias index + stable ids]
    C --> C2{look-alike names?}
    C2 -->|yes| C3[unresolved_alias finding — never merged]
    C2 -->|no| D[Group claims by canonical entity, sort by observed date]
    C3 --> D
    D --> E[Layer 1: normalized-text hash duplicates]
    E --> F[Layer 2: transition trigger on a shared anchor]
    F --> G[Layer 2b: concrete slot conflict, needs an observation order]
    G --> G2{both sides slot=value records?}
    G2 -->|yes, no order| G3[Layer 2c recheck: slot-equality decides]
    G2 -->|prose, no order| G4[Layer 2c hedge: likely_conflict, reason + confidence]
    G3 --> H{engine is embed?}
    G4 --> H
    G -->|ordered| H
    H -->|no| K[Report findings, each carrying its fidelity band]
    H -->|yes| I{Ollama reachable?}
    I -->|no| J[DEGRADED finding, fall back to heuristics]
    I -->|yes| L[Layer 3: bge-m3 similarity proposes, the recheck decides, composite confidence reported]
    L --> K
    J --> K
    K --> M[Exit code: 0 clean, 1 findings, 2 usage]

Eval

eval/scenarios.json holds 58 hand-written synthetic scenarios (no real or personal data): the 32 from v0.1.0, frozen with their original labels, the 16 written for v0.2.0, and the 10-scenario grey-zone batch written for v0.4.0 — labels first, engines second. These are the real measured numbers from this repo (Ollama bge-m3, 1024d):

What the grey-zone batch changed

Same 58 scenarios, same gold labels, engine before vs after the v0.4.0 recheck (the labels were committed before the engine change — the delta is not a relabelling):

What moved, and why:

• Grey-zone false negatives are gone. Four scenarios that both engines missed — canonical records disagreeing with no observation order (equal dates, one side undated, neither dated, a version bump with equal dates) — are now confirmed from slot = "value" equality alone. Heuristic contradicted_implicit recall 0.737 → 1.000, stale recall 0.714 → 0.857.

• Over-claiming is gone. The two prose-only candidates the embedding engine used to assert as contradicted_implicit are now reported as likely_conflict; ok precision is 1.000 on both engines and the hedged class scores 1.000/1.000.

• Paraphrase false positives stayed at 0 / 0 (5 tagged paraphrase scenarios, including the two v0.1.0 false positives re-created as gz-para-01/gz-para-02).

• The one remaining miss is the documented, frozen stale-05 label boundary — unchanged from v0.2.0 and deliberately not tuned away.

Against the v0.1.0 baseline, on the same 32 scenarios with the same gold labels:

Reproduce everything in one command (--strict exits 1 if a declared target is missed):

python eval/run_eval.py --strict

The protocol — gold labels written first, one run, no per-scenario tuning, and the one deliberate label disagreement kept visible — is documented in eval/README.md.

Ingest formats

Dates parse leniently on every supported interpreter (the tool ships its own fromisoformat polyfill; nothing relies on the 3.11 parser). Undated claims are still linted; undated and unsourced claims are flagged. Hostile input — empty files, malformed JSON, a file that is not UTF-8 — comes back as a bad_input finding instead of a traceback.

Alias resolution

Entity identity is built once at ingest (and reused by scan, reconcile, as-of and the admit write gate) from three sources, in decreasing order of how much they prove:

1. a declaration — (aka "Acme Corp") on an explicit line, or an aliases list on a JSON/JSONL record. An author said so;

2. the conservative normalized form — case, whitespace and punctuation collapse, then trailing legal-entity suffixes (Ltd, Inc, Corp, Company, LLC, GmbH, …) are dropped. acme and Acme Corp are one entity without anyone annotating it;

3. nothing else. Two names that merely look alike are never folded: a silent merge would fabricate a contradiction between two real entities (or erase a real one). Instead the pair comes back as unresolved_alias with its edit distance and similarity, for a human to settle.

$ memory-autopsy scan examples/aliases.md

unresolved_alias       L0 Acme Corp :: 'Acme Corp' (form 'acme') and 'Acma' (form 'acma') differ by 1 edit(s), similarity 0.75 — NOT merged; confirm whether they are one entity
contradicted_explicit  L2 Acme Corp examples/aliases.md:1 :: newer fact has transition trigger 'left' on shared anchor 'globex'
contradicted_implicit  L2 conf=1.0 billing examples/aliases.md:4 :: slot-equality: datastore 'self-hosted postgres 15 cluster' vs 'aurora mysql compatible cluster' (no observation order: undated and undated — the two canonical records disagree, neither is newer)
== 3 findings (contradicted_explicit=1, contradicted_implicit=1, unresolved_alias=1) | 5 facts | engine=heuristic

That one run shows all three rules at once: Acme Corp and acme were folded (first line and second line are one entity, so the transition fires at all), Acma was not folded despite being one edit away, and the two datastore records on lines 4–5 — which have no dates and therefore no observation order — are confirmed as a contradiction from slot = "value" equality alone. as-of reports the same ambiguity under unresolved_aliases, so a temporal read can never silently answer for two entities it could not tell apart.

The grey zone

Similarity proposes; structure disposes. Layer 3 (Ollama bge-m3) only proposes candidate pairs from BAND_GREY (0.45) upward — the class always comes from the deterministic recheck:

Every layer-3 finding reports a composite confidence — the cosine blended with how much the structure proved (canonical record 1.0, anchor/time cue 0.6, copula/prepositional frame 0.35). The pair below is the whole point of the layer: two dated prose claims whose frames barely overlap, so the heuristic engine refuses to pair them on a guess —

$ memory-autopsy scan examples/greyzone.md
clean: 2 facts, 0 findings (engine=heuristic)

$ memory-autopsy scan examples/greyzone.md --engine embed
likely_conflict        L3 score=0.661 conf=0.505 billing-store examples/greyzone.md:1 :: moderate claim similarity (0.66) with prose-only: slot 'f:billing' value conflict ('postgres' -> 'mysql'); composite confidence 0.51 (cosine 0.66 + structured 0.35)
== 1 findings (likely_conflict=1) | 2 facts | engine=embed

— and the embedding engine puts it in front of a human with both signals on the record, instead of either dropping it or declaring it a contradiction.

A hedge is a review item, not a verdict: likely_conflict is never planned as a fix, never rewritten by --apply, and never blocks a write on its own — the admit gate surfaces it as a warning instead.

Verdicts: reliability, capped by provenance

reconcile builds a per-entity verdict matrix. Conflicts rank by later observed_at, then higher source class, then capped reliability, then an explicit status verb, then source order.

Reliability is min(wording weight, source-class ceiling, fidelity ceiling):

we verified the billing store moved     -> 1.00  (primary)  -> 1.00
we verified the billing store moved     -> 1.00  (unclassified) -> 0.20
rumor has it the billing store moved    -> 0.20  (primary)  -> 0.20

The source-class ceiling is the provenance cap: when a primary-source claim exists for an entity, unclassified claims are removed from the ranking pool before anything is compared, so 40 padded junk sources cannot crowd out one documented fact. Capped claims are reported as CAPPED, not hidden.

$ memory-autopsy reconcile examples/memory.md examples/facts.json --registry examples/registry.yaml

The weight table lives in memory_autopsy/verdict.py and is overridable per project with --registry (a YAML subset — cues.<word>: <weight>, class.<class>: <weight>, fidelity.<band>: <weight>).

Reconcile output

# memory-autopsy reconcile

Sources: 2 | Entities: 9 | Claims considered: 15

## Globex

| Source | Claim | Observed | Status | Verdict |
|---|---|---|---|---|
| examples/memory.md | left Northwind | 2026-05-01 | — | WINNER |

**Canonical**: left Northwind  *(observed 2026-05-01 | class unclassified | reliability 0.20)*

Time travel (as-of)

"What was considered true on 2026-03-01?" — the payoff of keeping valid_from/valid_to around. A record with an explicit window is in effect at D when valid_from <= D < valid_to (the window is an assertion about the world, so it is trusted even when the record was observed later); a record with no window was believed from the moment it was observed (observed_at <= D); a record with neither is reported as unanchored instead of being silently folded into the answer. Conflicts among in-effect records resolve with the same provenance-cap rules the linter uses, so the two views of one store can never disagree.

The same store answers differently per date (real output, both commands run against examples/timeline.jsonl):

$ memory-autopsy as-of 2026-03-01 examples/timeline.jsonl

# memory-autopsy as-of 2026-03-01

Store: examples/timeline.jsonl | facts read: 7 | in effect: 4 | entities: 3
Conflicts resolved at this date: 1

## billing-store

| Slot | Value | Observed | Class | Reliability | Verdict |
|---|---|---|---|---|---|
| tool | PostgreSQL 14 | 2026-01-05 | primary | 0.70 | CURRENT |

## edge-proxy

| Slot | Value | Observed | Class | Reliability | Verdict |
|---|---|---|---|---|---|
| gateway | haproxy | 2026-02-01 | unclassified | 0.20 | CAPPED |
| gateway | kong | 2026-03-01 | primary | 0.70 | CURRENT |

- **CAPPED** gateway = 'haproxy' (id `c1ab7a22a4dcb381`) — unclassified claim, removed from the ranking by the primary-source fact `a7cbc633c4d1bdf6`

## nightly-sync

| Slot | Value | Observed | Class | Reliability | Verdict |
|---|---|---|---|---|---|
| schedule | 02:00 UTC | 2026-01-01 | primary | 0.70 | CURRENT |

## Not in effect on 2026-03-01

| Entity | Value | Reason |
|---|---|---|
| nightly-sync | 23:30 UTC | not_yet_valid |
| billing-store | MySQL 8 | not_yet_valid |

## Unanchored (no date, no validity window)

- Initech: 'grafana' (id `145a8166667a9e33`)

Four months later the same store flips: billing-store is now MySQL 8 (CURRENT) with PostgreSQL 14 listed under Not in effect with reason expired, and nightly-sync moved to 23:30 UTC.

$ memory-autopsy as-of 2026-07-01 examples/timeline.jsonl

# memory-autopsy as-of 2026-07-01

Store: examples/timeline.jsonl | facts read: 7 | in effect: 4 | entities: 3
Conflicts resolved at this date: 1

## billing-store

| Slot | Value | Observed | Class | Reliability | Verdict |
|---|---|---|---|---|---|
| tool | MySQL 8 | 2026-06-02 | primary | 0.70 | CURRENT |
...
## Not in effect on 2026-07-01

| Entity | Value | Reason |
|---|---|---|
| nightly-sync | 02:00 UTC | expired |
| billing-store | PostgreSQL 14 | expired |

--json returns the same payload machine-readably (beliefs, out_of_window with the reason per record, unanchored, facts_read, conflicts_resolved); --entity NAME narrows to one entity.

Undo

--apply rewrites files in place, so it snapshots them first — every mutated file gets a timestamped <name>.bak-<ts> copy next to it (the plain <name>.bak from earlier versions is still written too), the path is printed, and memory-autopsy undo <file> restores the newest snapshot. The round trip is byte-exact; a test asserts it (tests/test_undo.py).

$ memory-autopsy report memory.md --apply --yes-i-know

applied 4 fix(es) across 1 file(s):
  supersede memory.md:5 — "billing-store: uses PostgreSQL 14 for the billing store" (valid_to 2026-06-02)
    new: billing-store: tool = "PostgreSQL 14 for the billing store" ; valid_to=2026-06-02 -- 2026-01-05 (source: infra-doc)
  delete memory.md:11 — "cache-tier: uses redis 7" ([duplicate] identical to 'cache-tier: uses redis 7' (normalized hash match))
  supersede memory.md:15 — "works at Northwind" (valid_to 2026-05-01)
    new: Globex: employer = "Northwind" ; valid_to=2026-05-01 -- 2026-01-01 (source: profile)
  supersede memory.md:17 — "project Helios is active" (valid_to 2026-07-15)
    new: Umbra: project = "Helios is active" ; valid_to=2026-07-15 -- 2026-01-15 (source: roadmap)
  rewrote memory.md (backup: memory.md.bak, snapshot: memory.md.bak-20261017-090055-968964)
undo with: memory-autopsy undo <file>  (restores the newest snapshot)

$ memory-autopsy undo memory.md
restored memory.md from memory.md.bak-20261017-090055-968964

Snapshots accumulate (memory.md.bak-<first timestamp>, memory.md.bak-<second timestamp>, …) and sort chronologically, so undo always restores the newest one; the snapshots themselves are never deleted by the tool, and undo is idempotent rather than destructive. They are git-ignored (*.bak, *.bak-*).

Write gate (admit)

The SDK call for the write path: check one incoming fact before your agent persists it. admit is pure — no network, no writes — and returns a typed verdict with evidence, not just a boolean:

• accepted — safe to persist (the canonical 12-field record is included);

• requires_review — the fact disagrees with something already stored: every conflict carries the stable ids of the store facts involved, plus the supersede edit (stamp valid_to on the stored claim, never delete it) that would resolve the conflict;

• rejected — the fact cannot be normalized at all, or the store is unreadable (the gate never runs blind).

from memory_autopsy.api import admit

verdict = admit({"entity": "billing-store", "slot": "tool",
                 "value": "MariaDB", "date": "2026-07-01",
                 "source": "ops"}, "examples/store.jsonl")
print(verdict.status)          # requires_review
print(verdict.conflict_ids)    # ['8a0ccbab525e5078']
print(verdict.supersede)       # the exact edit that would resolve it

Real output of that call:

status: requires_review | reason: conflicts with 1 stored fact(s)
conflict_ids: ['8a0ccbab525e5078']
supersede: {
  "action": "supersede",
  "target_id": "8a0ccbab525e5078",
  "target_is_candidate": false,
  "target_shape": "jsonl",
  "valid_to": "2026-07-01",
  "new_line": "billing-store: tool = \"MySQL 8\" ; valid_to=2026-07-01 -- 2026-06-02 (source: infra-doc)",
  "reason": "[contradicted_implicit] slot 'anchor:uses' value conflict ('mysql' -> 'mariadb')"
}

Detection is the very same engine scan runs, so what the gate blocks and what the linter reports cannot drift apart. An idempotent re-admit of the identical fact is accepted (with already_present=True), and only active store facts can conflict — a claim already stamped valid_to is history and cannot block a new write.

Agent self-audit (MCP)

memory-autopsy-mcp (or python -m memory_autopsy.mcp_tool) speaks newline-delimited JSON-RPC over stdio — the dialect MCP clients use for local servers. Two tools, both read-only: the server never writes a store and never makes a network call; stdout carries only protocol responses.

Register it with any MCP client, for example Claude Desktop:

{
  "mcpServers": {
    "memory-autopsy": {
      "command": "memory-autopsy-mcp"
    }
  }
}

Your agent then calls memory_lint(store) before trusting a memory file, and memory_admit(store, fact) before writing one — the same gate and the same engine as the CLI, so an in-loop check and an offline audit can never disagree. Real round-trip (spawn, initialize, tools/list, tools/call):

$ initialize -> {"name": "memory-autopsy", "version": "0.4.0"}
$ tools/list -> ['memory_lint', 'memory_admit']
$ tools/call memory_lint(examples/store.jsonl) -> finding_count: 2 | counts: {"contradicted_explicit": 1, "stale": 1}

Threat model

Short version: the provenance cap defends against padded volumetric poisoning in a test harness — it is a linter, not a security control, and it does not authenticate sources. A forged primary-source claim wins outright. docs/threat-model.md states what is defended, what is not, and how each claim is tested.

Install

pip install .          # or: pipx install .
# run from source without installing:
PYTHONPATH=src python -m memory_autopsy.cli scan examples/memory.md

On Arch/CachyOS-style systems with an external-managed python (PEP 668), pip refuses to install into the system interpreter — use pipx or a venv:

pipx install .
# or
python -m venv .venv && .venv/bin/pip install -e .

The embedding layer needs a local Ollama with bge-m3 (ollama pull bge-m3). Without it, use --engine heuristic (the default) — fully offline.

Usage

# lint memory files (heuristic engine, offline)
memory-autopsy scan MEMORY.md FACTS.json STORE.jsonl

# add the embedding layer for implicit contradictions
memory-autopsy scan MEMORY.md --engine embed

# machine-readable, for cron/CI (every finding carries its reliability)
memory-autopsy scan MEMORY.md --json | jq '.counts'

# per-entity verdict matrix across sources, with the provenance cap applied
memory-autopsy reconcile snapshot-a.md snapshot-b.md --registry registry.yaml

# report + fix plan (dry-run by default)
memory-autopsy report MEMORY.md
memory-autopsy report MEMORY.md --apply --yes-i-know   # snapshots, then rewrites
memory-autopsy undo MEMORY.md                          # restore the newest snapshot

# time travel: what was considered true on a given date
memory-autopsy as-of 2026-03-01 STORE.jsonl
memory-autopsy as-of 2026-07-01 STORE.jsonl --entity billing-store --json

# MCP server for agent self-audit (stdio JSON-RPC)
memory-autopsy-mcp

Env: OLLAMA_BASE_URL (default http://localhost:11434), EMBED_MODEL (default bge-m3).

Troubleshooting cookbook

Roadmap

• Baseline mode: --baseline to record current findings and fail only on new ones (like ESLint's baseline workflow).

• Direct adapters for common agent memory stores (SQLite memory backends).

• Cross-format supersede: stamp valid_to back into a JSONL store line instead of only rewriting markdown.

• Project rule file (memory-autopsy.toml) for site-specific anchors and transitions, alongside the existing --registry reliability overrides.

Tests

python -m unittest discover -s tests -t .

206 stdlib-only unit tests, including the Ollama-down fallback (wrong port), the 40-source poisoning case, alias and fidelity matrices, degraded-input handling, supersede rewriting, the temporal-read (as-of) matrix, the admit write-gate matrix, a live MCP stdio round-trip (spawn, initialize, tools/list, tools/call, terminate), and a Python 3.9 syntax/API guard that runs on every source file. v0.4.0 adds three matrices:

• tests/test_alias.py — the alias matrix (declared aliases, conservative normalized forms, Ltd/Inc/Corp/Company suffixes) and the ambiguity guard: look-alike names are reported, never merged, by both scan and as-of;

• tests/test_greyzone.py — the verify-and-confirm contract: canonical records decide without an observation order, prose is hedged, a hedge is never auto-fixed and never blocks a write, and the composite confidence is the documented blend of both signals;

• tests/test_undo.py — apply → undo round trip is byte-exact for markdown and JSON stores, the newest snapshot wins, and undo on a file that was never applied exits 2.

License

MIT — see LICENSE. Copyright (c) 2026 Brodewa.