# journal-rag

Source-control-friendly **hybrid retrieval** over team markdown journals. Heading-chunked **BM25 + local vector embeddings** fused via Reciprocal Rank Fusion (RRF), with regex as an escape hatch. A per-user daemon shares models and workspace indexes across MCP clients.

Embeddings run locally via `@huggingface/transformers` — no API keys or hosted inference calls.

Each consuming repo commits `journal-rag.config.json` and markdown under `docs/journal/` (or other configured folders). This package is the shared engine.

## Per-repo config

Create `journal-rag.config.json` at the repo root:

```json
{
  "sources": ["docs/journal"],
  "cachePath": ".journal-rag/index.json"
}
```

| Field | Required | Default | Description |
|-------|----------|---------|-------------|
| `sources` | yes | — | Directories containing markdown journals |
| `cachePath` | no | `.journal-rag/index.json` | BM25 chunk index cache path |
| `embeddingModel` | no | Platform-specific model | Hugging Face model ID for local embeddings. macOS uses the cache-free `Xenova/all-MiniLM-L6-v2` encoder; other platforms use `onnx-community/Qwen3-Embedding-0.6B-ONNX`. |

The vector cache (`vectors.json`) is stored in the same directory as `cachePath`.

Add to `.gitignore`:

```
.journal-rag/
```

## Build & install (once per machine)

```bash
cd c:/repos/journal-rag
npm install          # runs prepare → build
npm link             # puts journal + journal-mcp on your PATH
```

`npm link` registers two global commands:

| Command | What it runs |
|---------|----------------|
| `journal` | CLI (`search`, `list`, `get`, …) |
| `journal-mcp` | MCP stdio server (for editor config) |
| `journal-daemon` | Shared local daemon (normally started automatically) |

Re-run `npm run build` (or `npm link` again) after pulling server changes.
Alternative to link: `npm install -g .` from this repo (same effect).

## CLI (any teammate)

From a repo root with config:

```bash
journal search "HttpFacade singleton"        # hybrid BM25 + vector (default)
journal search "HttpFacade singleton" --bm25 # BM25-only (no embedding)
journal list --filter dialog
journal get docs/journal/2026-04-21_vapp-http-facade-and-singleton-sweep.md
journal index --rebuild
```

After `npm link` in this repo, `journal search "..."` works globally.

Set `JOURNAL_RAG_WORKSPACE` to an absolute repo root only when you must run the CLI from a subdirectory.

The first run downloads a local embedding model (~90 MB on macOS and ~614 MB elsewhere) to the Hugging Face cache directory. Subsequent runs load from cache. Embedding inference uses DirectML on Windows, Core ML on macOS, and CPU elsewhere. macOS uses a cache-free encoder model because Core ML cannot execute the zero-length key/value-cache inputs used by the Qwen ONNX Community export, and cache-free Qwen exports consume several gigabytes when compiled by Core ML. If an accelerated provider fails during model loading or inference, the error is logged and that model is retried on CPU for the lifetime of the daemon. Indexing uses single-document batches to stay within the memory limits of 6 GB GPUs. DirectML sessions disable memory-pattern optimization and use sequential execution as required by ONNX Runtime. Partial vectors are checkpointed every 100 chunks so an interrupted build can resume.

Each editor still launches a small stdio MCP proxy because stdio cannot be shared between clients. The proxy automatically connects to one per-user daemon over a Windows named pipe or Unix socket. The daemon canonicalizes the path to `journal-rag.config.json`; Cursor, ChatGPT, and other clients using the same config therefore share one workspace runtime, vector build, and model instance. No administrator-level service installation is required.

## Diagnostic logs

The MCP proxy and daemon write daily JSONL logs and retain them for 14 days:

- Windows: `%LOCALAPPDATA%\journal-rag\logs`
- macOS: `~/Library/Logs/journal-rag`
- Linux: `$XDG_STATE_HOME/journal-rag/logs` or `~/.local/state/journal-rag/logs`

Set `JOURNAL_RAG_LOG_DIR` to override the directory. Logs include process IDs, canonical config paths, daemon startup and request failures, vector-build timing, selected execution device, model-load timing, and process memory after model loading. Query text and journal content are not logged.

## MCP tools

| Tool | Purpose |
|------|---------|
| `search_journal` | Hybrid BM25 + vector search with RRF fusion (`query`, `k`). Falls back to BM25-only if vector index is unavailable. |
| `write_entry` | Create a new journal entry (`title`, `content`). Auto-generates dated filename, incrementally updates the vector index. |
| `get_entry` | Full file by path or filename |
| `list_entries` | Browse metadata (`filter` optional) |
| `search_regex` | Exact / path / symbol lookup |

## Editor setup

Use **stdio** — spawn Node with `dist/server.js`.

### Put MCP config in the workspace, not your user profile

The server resolves `journal-rag.config.json` by walking up from its **working directory**. That file lives at each **consuming repo's root** (next to `docs/journal/`), not in `journal-rag` itself.

If you add the server to a **global / user-level** editor profile, the spawn cwd is usually wrong (home dir, editor install dir, last random folder, etc.) and the server cannot find config — even if you hardcode `"cwd": "C:/repos/my-repo"`, that breaks the moment you open a second repo workspace.

**Do this instead:** commit workspace-level MCP config **inside each repo** that has journals. Teammates run `npm link` once (see above) so `journal-mcp` is on PATH — no machine-specific paths in the committed JSON.

### Cursor

`.cursor/mcp.json` at the **repo root** (e.g. `my-repo/.cursor/mcp.json`) — safe to commit:

```json
{
  "mcpServers": {
    "journal": {
      "command": "journal-mcp",
      "cwd": "${workspaceFolder}",
      "env": {
        "JOURNAL_RAG_WORKSPACE": "${workspaceFolder}"
      }
    }
  }
}
```

`${workspaceFolder}` resolves to the repo you opened. `journal-mcp` comes from `npm link` in the `journal-rag` repo.

### VS Code (Copilot agent mode)

Same idea: `.vscode/mcp.json` in the repo, not User settings:

```json
{
  "servers": {
    "journal": {
      "type": "stdio",
      "command": "journal-mcp",
      "cwd": "${workspaceFolder}"
    }
  }
}
```

### JetBrains AI Assistant / Junie

Configure MCP at **project** scope (`.idea` / project settings), not the IDE default profile. Open the repo as the project root. Command: `journal-mcp` (after `npm link`).

### If `journal-mcp` is not found

Ensure npm's global bin dir is on your PATH (`npm bin -g`). On Windows that is usually `%APPDATA%\\npm`. Then re-run `npm link` from `journal-rag`. Fallback for a single machine only: `"command": "node", "args": ["<absolute-path>/journal-rag/dist/server.js"]`.

### Fallback

If an editor cannot set cwd per workspace, set env `JOURNAL_RAG_WORKSPACE` to the absolute path of the consuming repo root in that workspace's MCP config.

## Design notes

- Corpus is small (~tens of files); BM25 over heading chunks matches how journals are written.
- Vector embeddings (local, via Transformers.js) add semantic recall for paraphrased or conceptual queries.
- Reciprocal Rank Fusion (RRF, k=60) merges BM25 and vector rankings without needing score normalization.
- Index caches are optional and gitignored; markdown in git is the source of truth.
- Vector cache is incremental — only new/changed chunks are re-embedded on rebuild.
- Vector index builds take a machine-wide lock, so MCP servers opened for different workspaces do not run expensive embedding jobs concurrently. The lock has a heartbeat and is recovered after a crashed or killed process.
- The daemon owns workspace runtimes keyed by canonical config path. MCP stdio processes contain no model or index state and only forward tool calls.