journal-rag

Source-control-friendly hybrid retrieval over team markdown journals. Heading-chunked BM25 + local vector embeddings fused via Reciprocal Rank Fusion (RRF), with regex as an escape hatch. A per-user daemon shares models and workspace indexes across MCP clients.

Embeddings run locally via @huggingface/transformers (default model: Qwen3-Embedding-0.6B) — no API keys, no external calls.

Each consuming repo commits journal-rag.config.json and markdown under docs/journal/ (or other configured folders). This package is the shared engine.

Per-repo config

Create journal-rag.config.json at the repo root:

{
  "sources": ["docs/journal"],
  "cachePath": ".journal-rag/index.json",
  "embeddingModel": "onnx-community/Qwen3-Embedding-0.6B-ONNX"
}

The vector cache (vectors.json) is stored in the same directory as cachePath.

Add to .gitignore:

.journal-rag/

Build & install (once per machine)

cd c:/repos/journal-rag
npm install          # runs prepare → build
npm link             # puts journal + journal-mcp on your PATH

npm link registers two global commands:

Re-run npm run build (or npm link again) after pulling server changes. Alternative to link: npm install -g . from this repo (same effect).

CLI (any teammate)

From a repo root with config:

journal search "HttpFacade singleton"        # hybrid BM25 + vector (default)
journal search "HttpFacade singleton" --bm25 # BM25-only (no embedding)
journal list --filter dialog
journal get docs/journal/2026-04-21_vapp-http-facade-and-singleton-sweep.md
journal index --rebuild

After npm link in this repo, journal search "..." works globally.

Set JOURNAL_RAG_WORKSPACE to an absolute repo root only when you must run the CLI from a subdirectory.

The first run downloads the embedding model (~614 MB, quantized int8) to the Hugging Face cache directory. Subsequent runs load from cache. Embedding inference uses DirectML on Windows, Core ML on macOS, and CPU elsewhere. Indexing uses single-document batches to stay within the memory limits of 6 GB GPUs. DirectML sessions disable memory-pattern optimization and use sequential execution as required by ONNX Runtime. Partial vectors are checkpointed every 100 chunks so an interrupted build can resume.

Each editor still launches a small stdio MCP proxy because stdio cannot be shared between clients. The proxy automatically connects to one per-user daemon over a Windows named pipe or Unix socket. The daemon canonicalizes the path to journal-rag.config.json; Cursor, ChatGPT, and other clients using the same config therefore share one workspace runtime, vector build, and model instance. No administrator-level service installation is required.

Diagnostic logs

The MCP proxy and daemon write daily JSONL logs and retain them for 14 days:

• Windows: %LOCALAPPDATA%\journal-rag\logs

• macOS: ~/Library/Logs/journal-rag

• Linux: $XDG_STATE_HOME/journal-rag/logs or ~/.local/state/journal-rag/logs

Set JOURNAL_RAG_LOG_DIR to override the directory. Logs include process IDs, canonical config paths, daemon startup and request failures, vector-build timing, selected execution device, model-load timing, and process memory after model loading. Query text and journal content are not logged.

MCP tools

Editor setup

Use stdio — spawn Node with dist/server.js.

Put MCP config in the workspace, not your user profile

The server resolves journal-rag.config.json by walking up from its working directory. That file lives at each consuming repo's root (next to docs/journal/), not in journal-rag itself.

If you add the server to a global / user-level editor profile, the spawn cwd is usually wrong (home dir, editor install dir, last random folder, etc.) and the server cannot find config — even if you hardcode "cwd": "C:/repos/my-repo", that breaks the moment you open a second repo workspace.

Do this instead: commit workspace-level MCP config inside each repo that has journals. Teammates run npm link once (see above) so journal-mcp is on PATH — no machine-specific paths in the committed JSON.

Cursor

.cursor/mcp.json at the repo root (e.g. my-repo/.cursor/mcp.json) — safe to commit:

{
  "mcpServers": {
    "journal": {
      "command": "journal-mcp",
      "cwd": "${workspaceFolder}",
      "env": {
        "JOURNAL_RAG_WORKSPACE": "${workspaceFolder}"
      }
    }
  }
}

${workspaceFolder} resolves to the repo you opened. journal-mcp comes from npm link in the journal-rag repo.

VS Code (Copilot agent mode)

Same idea: .vscode/mcp.json in the repo, not User settings:

{
  "servers": {
    "journal": {
      "type": "stdio",
      "command": "journal-mcp",
      "cwd": "${workspaceFolder}"
    }
  }
}

JetBrains AI Assistant / Junie

Configure MCP at project scope (.idea / project settings), not the IDE default profile. Open the repo as the project root. Command: journal-mcp (after npm link).

If journal-mcp is not found

Ensure npm's global bin dir is on your PATH (npm bin -g). On Windows that is usually %APPDATA%\\npm. Then re-run npm link from journal-rag. Fallback for a single machine only: "command": "node", "args": ["<absolute-path>/journal-rag/dist/server.js"].

Fallback

If an editor cannot set cwd per workspace, set env JOURNAL_RAG_WORKSPACE to the absolute path of the consuming repo root in that workspace's MCP config.

Design notes

• Corpus is small (~tens of files); BM25 over heading chunks matches how journals are written.

• Vector embeddings (local, via Transformers.js) add semantic recall for paraphrased or conceptual queries.

• Reciprocal Rank Fusion (RRF, k=60) merges BM25 and vector rankings without needing score normalization.

• Index caches are optional and gitignored; markdown in git is the source of truth.

• Vector cache is incremental — only new/changed chunks are re-embedded on rebuild.

• Vector index builds take a machine-wide lock, so MCP servers opened for different workspaces do not run expensive embedding jobs concurrently. The lock has a heartbeat and is recovered after a crashed or killed process.

• The daemon owns workspace runtimes keyed by canonical config path. MCP stdio processes contain no model or index state and only forward tool calls.