# OpenCode Brain — Tier 3 Persistent Memory System

A local A-MEM–style agentic memory system for OpenCode with noise filtering,
selective forgetting, and iterative retrieval. Built on your existing stack.

**Stack:**
- **Qdrant** — vector store (you already have this)
- **sentence-transformers/all-MiniLM-L6-v2** — 384-dim embeddings (you already have this)
- **NetworkX** — knowledge graph, persisted to JSON on disk
- **Qwen3-4B / Qwen3-30B** — routed note enrichment, link decisions, memory evolution
- **Obsidian Local REST API** — human-readable vault mirror (optional)
- **FastMCP / MCP** — exposes everything as an MCP server to OpenCode

> Qdrant client compatibility: OpenCode Brain supports both legacy `search(...)`
> and modern `query_points(...)` APIs via an internal adapter.

---

## Architecture

```
OpenCode ──MCP──► mcp_server.py (18 tools)
                        │
        ┌───────────────┼──────────────────┐
        ▼               ▼                  ▼
    Qdrant          NetworkX           Obsidian
 (vectors +       (link graph,        (markdown
  payloads,        typed edges,        mirror,
  access_count,    JSON on disk)       optional)
  last_accessed)
        │               │
        └───────┬────────┘
                ▼
      Qwen3 router
    (note construction,
     link decisions,
     memory evolution,
     context distillation)
```

### Three Core Improvements over Basic A-MEM

| Improvement | Module | Research Basis |
|---|---|---|
| Noise filtering via distillation | `brain/distiller.py` | MEM1 (Zhou et al., 2025): agents that discard irrelevant info outperform by 3.5x |
| Selective forgetting / decay | `brain/decay.py` | SimpleMem (Liu et al., 2026): selective retention achieves 26.4% F1 improvement |
| Iterative retrieval | `brain/deep_search.py` | Structural Memory (Zeng et al., 2024): iterative retrieval outperforms single-step across all benchmarks |

---

## Memory Pipeline

### `add_memory` (permanent note)
1. **Note Construction** — routed LLM generates title, context, keywords, tags
2. **Embedding** — `all-MiniLM-L6-v2` encodes `content + context` → 384-dim vector
3. **Upsert** — stored in Qdrant with `access_count=0`, `last_accessed=""`
4. **Link Generation** — top-K semantic candidates fetched; routed LLM decides which to link
5. **Memory Evolution** — for high-similarity links, hard model re-generates old notes' context; old vectors updated. Runs in a **background thread** so `add_memory` returns immediately.
6. **Graph Update** — NetworkX edges added with typed relations; mutations are **batched** and flushed once, not per-edge
7. **Obsidian Sync** — markdown written to vault (best-effort, non-blocking)

### `add_memory` (fleeting note)
Steps 1–3 only. **Link generation and memory evolution are skipped** — fleeting notes
are cheap quick captures. Hard LLM calls happen at distillation time, not capture time.
Fleeting notes are **excluded from all search results by default**.

### `distill_to_permanent`
Runs the full permanent pipeline on a fleeting note: re-enrichment → re-embedding → link
generation → memory evolution → Obsidian sync. Promotes `memory_type` to `"permanent"`.

### `deep_search` (iterative retrieval)
```
Hop 0: embed(query) → search Qdrant → [A, B, C]
       stamp_access(A, B, C)  ← decay tracking
Hop 1: embed(query + titles of A,B,C) → search Qdrant → [D, E, F]
+ graph.get_links(A,B,C) → pre-fetch neighbours
stamp_access(neighbours) ← graph-expanded notes are also tracked for decay
Hop 2: embed(query + titles of D,E,F) → search Qdrant → [G, H]
Final: union of all hops, ranked by best score seen across hops
```

### Decay (selective forgetting)
Every `search_notes` call stamps `last_accessed` and increments `access_count` on
returned notes. Graph-expanded neighbors surfaced by `deep_search` are **also stamped**
(BUG-4 fix — previously only direct vector-search hits were tracked).

`archive_stale_notes` soft-archives notes that have:
- Not been accessed in > 45 days, AND
- Fewer than 2 lifetime accesses, AND
- `decay_score` < 0.3 (recency 60% + access importance 40%)

`archive_stale_notes` paginates through **all notes** regardless of vault size
(pages through the full collection — no hidden cap).

Archived notes (`memory_type="archived"`) vanish from all searches but are never deleted.
`restore_note(zk_id)` reverses archival instantly.

---

## Payload Schema (Qdrant)

Every point stored in Qdrant has these fields:

| Field | Type | Description |
|---|---|---|
| `zk_id` | str | `"ZK-<8-char-hex>"` — unique note ID |
| `title` | str | Single atomic claim (LLM-generated) |
| `content` | str | Raw knowledge content |
| `context` | str | Why this matters — LLM-generated, evolves over time |
| `keywords` | list[str] | 5–10 searchable terms (LLM-generated) |
| `tags` | list[str] | 2–5 category tags |
| `links` | list[str] | zk_ids of linked notes (bidirectional) |
| `memory_type` | str | `fleeting` \| `permanent` \| `archived` |
| `note_kind` | str | `mistake` \| `fix` \| `pattern` \| `research` \| `decision` \| `general` \| `open_thread` \| `session_summary` \| `project_context` |
| `pinned` | bool | Protect high-value notes from automatic archival |
| `project` | str | `rag-chatbot` \| `resume-screener` \| `nano-r1` \| `general` |
| `created` | str | ISO-8601 timestamp |
| `updated` | str | ISO-8601 timestamp |
| `access_count` | int | Lifetime search retrieval count (decay tracking) |
| `last_accessed` | str | ISO-8601 timestamp of last search hit (decay tracking) |

---

## MCP Tools Reference (18 tools)

| Tool | When to Use |
|---|---|
| `open_session(project)` | Preferred one-call session bootstrap; wraps `get_session_context` plus `get_inbox` |
| `get_session_context(project)` | Canonical session-start fallback; loads full project state |
| `get_inbox(project)` | Canonical second fallback call; surfaces fleeting notes for distillation |
| `distill_to_permanent(zk_id)` | Promote a fleeting note to permanent knowledge |
| `deep_search(query, project, hops, include_contextual=false)` | **Complex questions** — multi-hop iterative retrieval |
| `search_memory(query, top_k, project, include_contextual=false)` | Simple direct lookups only |
| `get_related(zk_id, depth)` | Follow graph links from a specific note |
| `add_memory(content, project, memory_type, note_kind, ...)` | Store new knowledge |
| `update_memory(zk_id, new_content)` | Correct or extend an existing note |
| `archive_stale_notes(dry_run)` | Soft-archive unused notes (weekly maintenance) |
| `close_session(project, objective, completed, blockers, next_steps, key_files=None)` | Preferred one-call handoff capture; writes `session_summary` plus active `open_thread` |
| `brain_stats()` | Health check — counts by type, graph size, Qdrant status |
| `record_failure(content, project, failure_signature)` | Capture a fresh failure as a fleeting mistake note |
| `record_resolution(failure_zk_id, fix_content, project)` | Capture a validated fix linked to prior failure |
| `preflight_check(task, project, top_k)` | Retrieve similar past mistakes/fixes before coding |
| `restore_note(zk_id)` | Restore one archived note to permanent |
| `decay_report(limit)` | Inspect lowest-decay notes likely to go stale next |
| `consolidate_clusters(project, dry_run, selected_cluster_ids, ...)` | Manual cluster consolidation with dry-run and selected apply |

### Session Ergonomics

Use `open_session(project)` at the start of coding-agent work when available. It preserves the canonical `get_session_context(project)` and `get_inbox(project)` payloads while reducing session startup to one call.

Use `close_session(...)` at handoff. It writes one `session_summary`, writes one active `open_thread`, and archives older `open_thread` notes only after the new handoff is durable.

### When to use `deep_search` vs `search_memory`

```
Simple direct lookup    →  search_memory  ("what port does Qdrant use?")
Complex / causal        →  deep_search    ("why does the CrossEncoder slow things down?")
Session start context   →  open_session   (preferred; canonical fallback is get_session_context + get_inbox)
"How did we solve X?"   →  deep_search    (may span multiple sessions and notes)
After finding a note    →  get_related    (follow the knowledge graph)
```

---

## File Structure

```
opencode-brain/
├── brain/
│   ├── embedder.py       # sentence-transformers wrapper (singleton, normalised)
│   ├── vector_store.py   # all Qdrant ops: upsert, search, scroll, update, decay tracking
│   ├── graph.py          # NetworkX DiGraph: typed edges, JSON persistence
│   ├── note_builder.py   # routed LLM prompts: construct_note, decide_link, evolve_context
│   ├── model_router.py   # centralised LLM routing: fast/hard model selection
│   ├── memory_evolution.py  # A-MEM core: link generation + retroactive context updates
│   ├── obsidian_sync.py  # vault write-back via Local REST API (best-effort)
│   ├── distiller.py      # noise filtering: inbox management + fleeting→permanent promotion
│   ├── decay.py          # selective forgetting: decay scores + stale note archival
│   ├── deep_search.py    # iterative retrieval: search → graph expand → search again
│   ├── consolidation.py  # cluster detection and merge (manual-only, dry-run first)
│   ├── session.py        # session context, startup wrapper support, close handoff helpers
│   └── mistake_memory.py # mistake-aware helpers: build_failure_note, rank_preflight_results
├── mcp_server.py     # FastMCP entry point, all 18 tools defined here
├── config.py         # all configuration, all overridable via .env
├── requirements.txt
├── .env.example      # copy to .env and fill in HUGGINGFACE_API_KEY at minimum
└── AGENTS.md         # brain protocol — paste into your OpenCode AGENTS.md
```

---

## Setup

### 1. Prerequisites

Qdrant must be running locally:
```bash
docker run -p 6333:6333 qdrant/qdrant
```

### 2. Install dependencies

```bash
cd opencode-brain
pip install -r requirements.txt --break-system-packages
```

Key dependencies:
- `qdrant-client>=1.13,<2.0` — constrained for predictable compatibility; supports both `search()` and `query_points()` APIs
- `huggingface-hub>=0.23.0` — HuggingFace Inference API for Qwen routing
- `sentence-transformers>=3.0.0` — local embedding model

### 3. Configure

```bash
cp .env.example .env
# Edit .env — set HUGGINGFACE_API_KEY at minimum
# Everything else has working defaults
```

### 4. Test the server runs

```bash
python mcp_server.py
# Should print:
#   [embedder] Loading sentence-transformers/all-MiniLM-L6-v2 …
#   [vector_store] Created collection 'opencode_brain'   (first run)
#   [graph] Loaded 0 nodes, 0 edges
# Then waits for stdio MCP input — Ctrl+C to exit
```

### 5. Add to OpenCode config

Edit `~/.config/opencode/config.json`:
```json
{
  "mcp": {
    "opencode-brain": {
      "command": [
"python",
            "C:\\Projects\\Brainn\\mcp_server.py"
      ],
      "type": "local"
    }
  }
}
```

### 6. (Optional) Enable Obsidian vault sync

1. Obsidian → Settings → Community Plugins → search **"Local REST API"** → Install → Enable
2. Copy the API key from the plugin settings page
3. Add to `.env`:
   ```
   OBSIDIAN_API_KEY=your-key-here
   OBSIDIAN_VAULT_SUBFOLDER=brain
   ```
Notes will be written to `<your-vault>/brain/zk/ZK-XXXXXXXX.md` with full YAML frontmatter.

### 7. Add the brain protocol to your AGENTS.md

Copy the contents of `AGENTS.md` (in this repo) into your existing OpenCode AGENTS.md.
The protocol defines exactly when each tool should be called during a session.

---

## Seed Your Brain (Recommended First Step)

Run once to pre-load your existing project knowledge:

```python
import sys
sys.path.insert(0, ".")
from brain import vector_store
from mcp_server import add_memory

vector_store.ensure_collection()

seeds = [
    {
        "content": (
            "RAG Chatbot stack: Qdrant hybrid search (dense + BM25) + CrossEncoder reranking "
            "+ LangGraph with SqliteSaver for persistent memory + Qwen3 routed models + "
            "all-mpnet-base-v2 embeddings + Chainlit UI. "
            "Location: C:/Projects/agentic-rag/. "
            "Known fix: QdrantClient shutdown ResourceWarning → atexit.register(client.close)."
        ),
        "project": "rag-chatbot",
        "memory_type": "permanent",
    },
    {
        "content": (
            "AI Resume Screener deployed to HuggingFace Spaces (Adityaladi/Ai_resume_screener). "
            "Stack: TF-IDF + Naive Bayes/KNN/SVC, 88-92% accuracy across 25 categories. "
            "Flask REST API + Streamlit frontend + Docker + GitHub Actions CI/CD. "
            "Known issue: IT category bias — fix with class_weight='balanced' or SMOTE."
        ),
        "project": "resume-screener",
        "memory_type": "permanent",
    },
    {
        "content": (
            "Nano-R1: QLoRA/GRPO fine-tune of Qwen2.5-3B-Instruct replicating DeepSeek-R1 "
            "chain-of-thought reasoning. Trained via Unsloth/TRL targeting GSM8K math. "
            "Published at HuggingFace: Adityaladi/Nano-R1. "
            "Remaining work: GSM8K evaluation via lm-evaluation-harness to close the metric gap."
        ),
        "project": "nano-r1",
        "memory_type": "permanent",
    },
    {
        "content": (
            "OpenCode MCP config pattern: 'type: remote' silently fails for remote MCP servers. "
            "Correct pattern: type='local', command='npx', args=['mcp-remote', '<url>']. "
            "Applies to Consensus, Exa, Context7, HuggingFace servers."
        ),
        "project": "general",
        "memory_type": "permanent",
    },
]

for seed in seeds:
    result = add_memory(
        seed["content"],
        project=seed["project"],
        memory_type=seed["memory_type"],
    )
    print(result)
```

---

## Weekly Maintenance

```python
# Check what is going stale (dry run — safe)
result = archive_stale_notes(dry_run=True)

# Apply if the list looks reasonable
result = archive_stale_notes(dry_run=False)

# Health check
print(brain_stats())
```

---

## Schema Conformance and Migrations

Run these in order when upgrading or validating data integrity:

```bash
# 1) Lifecycle/semantics split migration (strict scope)
python tools/migrate_memory_schema.py
python tools/migrate_memory_schema.py --apply

# 2) note_kind enum conformance migration
python tools/migrate_note_kind_conformance.py
python tools/migrate_note_kind_conformance.py --apply

# 3) memory_type enum conformance for semantic-type drift (for example memory_type="research")
python tools/migrate_memory_type_conformance.py
python tools/migrate_memory_type_conformance.py --apply

# 4) CI-safe schema drift check (non-zero exit if invalid payloads exist)
python tools/check_schema_health.py
```

Backups are written before `--apply` runs:
- `backups/memory-pre-migration-<timestamp>.json`
- `backups/notekind-pre-migration-<timestamp>.json`
- `backups/memorytype-pre-migration-<timestamp>.json`

---

## Tunable Config Values (`.env`)

| Variable | Default | Description |
|---|---|---|
| `HUGGINGFACE_API_KEY` | — | **Required** |
| `FAST_MODEL_ID` | `Qwen/Qwen3-4B-Instruct-2507` | Fast default LLM for note construction and extraction |
| `FAST_MODEL_PROVIDER` | `featherless-ai` | Inference provider for the fast model (overridden to `nscale` in `.env`) |
| `HARD_MODEL_ID` | `Qwen/Qwen3-30B-A3B-Instruct-2507` | Hard fallback model for synthesis and memory evolution |
| `HARD_MODEL_PROVIDER` | `featherless-ai` | Inference provider for the hard model |
| `QDRANT_HOST` | `localhost` | Qdrant server host |
| `QDRANT_PORT` | `6333` | Qdrant server port |
| `QDRANT_COLLECTION` | `opencode_brain` | Collection name |
| `EMBEDDING_MODEL` | `sentence-transformers/all-MiniLM-L6-v2` | Embedding model |
| `OBSIDIAN_API_KEY` | _(empty — sync disabled)_ | Obsidian Local REST API key |
| `OBSIDIAN_HOST` | `http://localhost:27123` | Obsidian REST API URL |
| `OBSIDIAN_VAULT_SUBFOLDER` | `brain` | Subfolder in vault for notes |
| `STALE_DAYS_THRESHOLD` | `45` | Days before a note is stale |
| `STALE_ACCESS_THRESHOLD` | `2` | Min accesses to be immune from archival |
| `GRAPH_PATH` | `~/.opencode-brain/graph.json` | Knowledge graph persistence path |

---

## Research Basis

**Core Architecture**
- [A-MEM: Agentic Memory for LLM Agents](https://arxiv.org/abs/2502.12110) (Xu et al., 2025) — Zettelkasten-inspired atomic notes + semantic linking + memory evolution
- [Zettelkasten Method](https://zettelkasten.de/) — atomic notes, bidirectional links, claim-based titles

**Three Core Improvements**
- [MEM1: Learning to Synergize Memory and Reasoning](https://arxiv.org/abs/2506.15841) (Zhou et al., 2025) — noise filtering via distillation; agents that discard irrelevant info outperform by 3.5× (NeurIPS 2025)
- [SimpleMem: Efficient Lifelong Memory](https://arxiv.org/abs/2601.02553) (Liu et al., 2026) — selective forgetting; 26.4% F1 improvement, 30× token reduction
- [On the Structural Memory of LLM Agents](https://arxiv.org/abs/2412.15266) (Zeng et al., 2024) — iterative retrieval; consistently outperforms single-step across all benchmarks (HotPotQA 82.1% F1)

**Supporting Research**
- [FadeMem: Biologically-Inspired Forgetting for Efficient Agent Memory](https://arxiv.org/abs/2601.18642) (Wei et al., 2026) — direction for adaptive decay; importance-modulated decay rates
- [CTIM-Rover: Pitfalls of Episodic Memory in SE Agents](https://arxiv.org/abs/2505.23422) (Lindenbauer et al., 2025) — motivation for decaying short-lived contextual memory; episodic memory noise degrades retrieval without active filtering (REALM 2025)
- [Mistake Notebook Learning](https://arxiv.org/abs/2512.11485) (Su et al., 2025) — conceptual basis for `record_failure` / `record_resolution` typed mistake workflow