# lucidmem

A memory engine for AI agents, with **per-project isolation that actually
holds**, a dashboard to manage what's in it, and an MCP connector so agents can
use it directly.

Agents forget everything between sessions. The usual fix is a vector store, and
the usual problem is that a vector store has no idea who is asking — so the
moment more than one person or project uses it, everyone sees everything.
lucidmem makes the project (a **space**) the unit of access, and enforces it in
Postgres rather than in application code.

```
Space ──┬── Document ──── Memory (chunk)
        └── Fact (versioned; a contradiction supersedes its parent)
```

- **Two layers.** A *curated* layer of hand-written Markdown in git
  ([OKF v0.2](https://github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf)),
  and a *corpus* layer of ingested material. Curated outranks corpus, and the UI
  and API label which is which — because "someone wrote this down on purpose" and
  "this appeared in a log once" deserve different trust.
- **Explainable ranking.** Every result decomposes its own score into vector,
  lexical, recency, curated and rerank contributions. `explain_ranking` compares
  any two results.
- **Degrades honestly.** GPU services are optional. When they are gone, search
  still works and says `rerank: unavailable` rather than silently changing.
- **Multi-tenant by construction.** Row-level security is the boundary;
  application filtering is only the fast path.

## Install

```bash
git clone https://github.com/libr3andr3/lucidmem && cd lucidmem
cp deploy/env.example deploy/.env      # set POSTGRES_PASSWORD and KB_APP_PASSWORD
docker compose -f deploy/compose.yml --env-file deploy/.env up -d
```

Then create a project and a token:

```bash
kb init myproject          # scaffolds a bundle, kb.yaml, .mcp.json, an agent skill
kb apply                   # reconcile grants (prints a diff first)
kb token issue user:you
```

Open <http://127.0.0.1:8090> and paste the token.

For just the CLI and MCP server:

```bash
pip install -e .
```

## Connect an agent

**[docs/CONNECT.md](docs/CONNECT.md)** is the full guide. The short version:

```json
{
  "mcpServers": {
    "lucidmem": {
      "command": "python",
      "args": ["-m", "lucidmem.mcp_server"],
      "env": { "LUCIDMEM_API_URL": "https://kb.example.com" }
    }
  }
}
```

Tools: `find_memory`, `drill_down`, `save_memory`, `ingest_file`, `promote`,
`explain_ranking`, `fact_history`, `memory_stats`, `whoami`.

## Access control

| Concept | Meaning |
|---|---|
| **space** | A project. The unit of isolation. |
| **principal** | A user, agent or service. Holds tokens. |
| **grant** | principal-or-group × space × role |
| **role** | `reader` → `contributor` → `curator` → `admin` |

Grants live in a `kb.yaml` per project, so "who can read this?" is a reviewable
diff instead of tribal knowledge:

```yaml
space: sales
title: Sales
sources:
  - {path: ./contracts, parser: pdf, tags: [legal]}
groups:
  bizdev: ["user:ana", "user:ben"]
grants:
  - {principal: "user:ana", role: admin}
  - {group: bizdev, role: contributor}
```

`kb apply` reconciles it and asks before revoking anything.

**The boundary is Postgres RLS.** `kb-api` connects as a role that cannot bypass
it and sets the principal per transaction. Clients get a bearer token, never a
DSN. `tests/test_isolation.py` includes a test that bypasses the application
entirely and queries the table directly.

## Ingest

| Parser | Handles |
|---|---|
| `transcript` | Claude Code session `.jsonl` — keeps prompts, prose, commands, short results; drops thinking and file dumps (~1% of bytes on real data) |
| `pdf` | Text layer, else OCR, else a VLM description for pages that are one figure |
| `image` | Description plus transcription |
| `okf` | An OKF bundle — the curated layer |
| `text` | Markdown, code, config |

Credentials are stripped **before** anything is embedded or sent to a model, so
a secret pasted into a terminal two months ago never becomes a searchable
memory. Ingestion is a queue: GPU-backed work waits when the GPU is away instead
of failing.

## Optional GPU services

All optional, all degrade cleanly. `deploy/serve.sbatch` is an example SLURM job.

| Service | Used for | Without it |
|---|---|---|
| VLM (any OpenAI-compatible endpoint) | contextual-retrieval prefixes, fact extraction, figure description | prefixes skipped; chunks embed bare |
| OCR | scanned documents | those files stay queued |
| Embeddings (GPU) | bulk ingest | falls back to CPU (~15s/chunk vs ~0.1s) |
| Reranker | final ordering | plain RRF, reported in `why` |

The GPU and CPU embedders run the same code so their vectors share a space;
`tests/test_embed_drift.py` asserts mean cosine > 0.99.

## Tests

```bash
pytest                                            # units, no database
docker run -d --name kb-pgtest -e POSTGRES_PASSWORD=testpw \
  -e POSTGRES_USER=lucidmem -e POSTGRES_DB=lucidmem \
  -p 127.0.0.1:5433:5432 pgvector/pgvector:pg17
LUCIDMEM_TEST_DB=1 pytest                         # + the isolation suite
```

The isolation suite runs against a deterministic hash embedder, so it needs no
GPU. A security test that only runs when a GPU is free is a test that never runs.

## Docs

- [docs/CONNECT.md](docs/CONNECT.md) — connecting an agent
- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — how it fits together, and why

## License

MIT