# Context Lattice

Context Lattice is a standalone, source-verifiable memory index for AI coding-agent
conversation exports. Point it at JSON or JSONL produced by tools such as Codex, Cursor,
Kiro, or your own agent, then retrieve a compact evidence bundle instead of replaying an
entire history into the next context window.

Raw records remain immutable. Hierarchical summaries are lossy, disposable navigation
indexes. Every result points back to the source file, JSON record, and SHA-256 hash.

## Quick start

No API key or external database is required. Python 3.11+ is supported.

```bash
cd /path/to/context-lattice
python3 -m pip install -e .

context-lattice init --root ~/exports/coding-sessions/
context-lattice ingest ~/exports/coding-sessions/
context-lattice query "Why did we change the retry policy?"
context-lattice query "What is the latest payment reference?" --json --debug
context-lattice inspect
context-lattice verify
context-lattice doctor
```

By default the database is created at `~/.context-lattice/memory.db`. Override it with
`--db /path/to/memory.db`. Ingestion accepts a single `.json`, a `.jsonl`, or a directory
tree containing both. `init` prints ready-to-paste MCP configuration using the same database
and source roots, avoiding CLI/server configuration drift.

## Install as an MCP server

Context Lattice exposes `memory_search`, `memory_get`, `memory_explain`, `memory_sync`, and
`memory_status` over local stdio MCP. Configure the database and an explicit source
allowlist, then point any MCP-capable client at the installed executable:

```json
{
  "mcpServers": {
    "context-lattice": {
      "command": "context-lattice-mcp",
      "env": {
        "CONTEXT_LATTICE_DB": "/absolute/path/to/memory.db",
        "CONTEXT_LATTICE_ALLOWED_ROOTS": "/absolute/path/to/agent/sessions"
      }
    }
  }
}
```

Multiple allowed roots use the platform path separator (`:` on macOS/Linux, `;` on
Windows). `memory_sync` is the only mutating MCP operation. Search evidence is explicitly
marked untrusted, bounded by a token budget, and resolvable to source hashes. See
[`docs/MCP.md`](docs/MCP.md).

## Input adapters

`--adapter auto` examines record envelopes and currently recognizes:

- `codex`: event JSONL containing `session_meta` and `response_item` records;
- `cursor`: exported conversations containing messages, bubbles or turns;
- `kiro`: role/content JSONL with a metadata header;
- `claude`: Claude Code project JSONL with nested message blocks;
- `canonical`: the stable Context Lattice v1 format;
- `generic`: common role/content, speaker/text and nested-message shapes.

Provider formats can change. The Codex, Cursor and Kiro adapters are intentionally tolerant
and fixture-tested, but the canonical schema is the guaranteed integration boundary:
[`schemas/conversation-v1.schema.json`](schemas/conversation-v1.schema.json). See
[`examples/canonical.json`](examples/canonical.json) for the smallest complete example.

```json
{
  "schema": "context-lattice/v1",
  "conversation_id": "release-planning",
  "messages": [
    {
      "role": "user",
      "content": "The deployment region is ap-south-1.",
      "timestamp": "2026-08-20T10:00:00Z",
      "fact_key": "deployment-region",
      "entities": ["ap-south-1"]
    }
  ]
}
```

Records without recognizable conversational content are counted as skipped. Malformed or
failed records are reported in the import result and stored in the import audit log.
Re-importing the same file is idempotent.

## Retrieval model

- immutable, append-only raw events in SQLite;
- original raw JSON plus file/record provenance and hashes;
- FTS5/BM25 for exact identifiers;
- an inverted sparse-postings index with an offline feature-hashing baseline;
- fixed-fanout chronological summary trees;
- reciprocal-rank fusion across lexical, semantic and hierarchical candidates;
- correction chains through optional `fact_key` values;
- disagreement-triggered search expansion and confidence-based abstention;
- explicit evidence-token budgets and inspectable retrieval traces.

The indexer and retriever are deterministic and make no LLM calls. The bundled semantic
model is feature hashing, so it is portable and exact-repeatable but weaker than a learned
embedding model. An external LLM may consume the evidence; it is not trusted to maintain
the memory index. The embedder boundary can be replaced without changing the evidence
contract.

The chronological hierarchy is a segment-tree-like navigation index. It cannot replace
semantic or lexical lookup: trees prune time ranges, while FTS5 and sparse postings locate
terms and concepts. Query-time dot products are aggregated inside SQLite, and only a
bounded root set and beam descend the tree. See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).

## Test and evaluate

```bash
python3 -m unittest discover -s tests -v
python3 -m context_lattice.cli eval --output benchmark-results.json
python3 -m context_lattice.cli golden-eval --output golden-results.json
python3 -m context_lattice.cli demo \
  "What is the current meeting room for team-3?"
```

The deterministic 50-question evaluation compares a recent-token window, rolling summary,
flat vector search and hierarchical hybrid retrieval. Its `answer_accuracy` is an evidence
sufficiency metric—not an LLM-judge score. See [`benchmark-results.json`](benchmark-results.json)
and [`DESIGN.md`](DESIGN.md).

The manually reviewed golden-v1 suite is the release gate for source recall, precision,
ranking, stale facts, unsupported results, citation integrity, and hierarchy branch recall.
It intentionally fails the command when thresholds regress. See
[`docs/GOLDEN_EVAL.md`](docs/GOLDEN_EVAL.md).

## Production posture

The local-first core has atomic per-conversation index updates, WAL concurrency,
cross-process maintenance locking, immutable events, source verification, bounded inputs and
retrieval, nested-symlink-safe MCP allowlisting, schema compatibility checks, CI, and
deterministic release gates. `context-lattice doctor` checks database integrity, index
freshness, FTS5, permissions, SQLite, Python, and MCP. Provider formats remain unofficial and
can change; the canonical v1 schema is the stable integration boundary. Review
[`SECURITY.md`](SECURITY.md) before exposing anything beyond local stdio.
The concrete release checklist and current non-goals are in
[`docs/RELEASE_GATES.md`](docs/RELEASE_GATES.md).