roam-semantic-search

Fully local semantic search over a Roam Research graph: fetch clear-text content through the Roam Local API, embed it with a locally hosted model, store vectors in a single SQLite file, and answer meaning-based queries from a CLI or an MCP server. Nothing about the graph's content ever leaves the machine — that constraint is the project's founding requirement, and it is enforced in code: the embedding client refuses any non-loopback server URL.

Roam Desktop ──(Local API, localhost HTTP)──► fetch ──► normalize ──► embed ──► store
                                                                        ▲          │
                                                          Ollama (localhost)   SQLite (FTS5 + vector blobs)
                                                                                   │
                                              MCP server (stdio) ◄── query ◄───────┘
                                              CLI (roam-semantic-search search)

Full design, phase results, and decision log: docs/design-plan.md.

How it works

• Fetch — one flat Datalog pull of every entity carrying a :block/uid (pages and blocks alike) through the Roam Local API; ~1 s for a 10k-entity graph. For an encrypted graph, the running Roam Desktop client is the only clear-text doorway, so the indexer runs on the same machine.

• Normalize — each block embeds with its breadcrumb: the page title plus ancestor block texts, root-first (ordered by ancestor count, never by wire order, which is creation order and diverges from depth on ~12% of nested blocks). Roam markup is cleaned to prose ([[refs]] → text, ((uid)) references resolve to their target's text one level deep); roam/js and roam/css pages are skipped, and daily-note pages are indexed (skippable with --no-daily-notes). Each record also carries retrieval emphasis in three weight tiers: the page names its own text references ([[Page]] and #tag alike — its concepts, highest), its direct-child tags:: values (its tags, middle), and its plain words plus its whole subtree's folded text (base). The keyword leg realizes the tiers as per-column BM25 weights (4/2/1); the vector leg by embed-input composition (labeled concept/tag segments, descendant text truncated first).

• Embed — a local Ollama server running nomic-embed-text (768-dim), with the model's search_document: / search_query: retrieval prefixes. Loopback-only, enforced.

• Store — one SQLite file (default ~/.cache/roam-semantic-search/<graph>.db): records + float32 embedding blobs, an FTS5 keyword mirror, and provenance meta. No SQLite extensions; vector KNN is a brute-force numpy matrix product (milliseconds at this scale).

• Query — hybrid retrieval: cosine KNN and BM25 rankings fused by reciprocal rank fusion, so paraphrase ("where do I argue…") and exact identifiers both rank.

• Refresh — incremental: re-fetch + re-normalize everything (cheap), then re-embed only records whose content hash changed and delete vanished uids. Selection is by content hash alone — an edit changes descendants' breadcrumbs and referrers' resolved text, which no per-entity timestamp can see. A no-change refresh takes ~2 s.

Related MCP server: ragi

Requirements

• Roam Desktop running locally with the Local API enabled, and each graph you want to index connected (Roam records the connection; see Graphs below)

• Ollama with the embedding model pulled: ollama pull nomic-embed-text (brew services start ollama keeps it running at login)

• Python ≥ 3.14 and a sibling checkout of guffin (the Local API transport layer)

Install

python3.14 -m venv .venv
.venv/bin/pip install -e ../guffin
.venv/bin/pip install -e ".[dev]"

Graphs

Several Roam graphs are typically connected at once, so every command names the graph it operates on. A graph is named by the nickname it was connected under, or by its canonical Roam name:

roam-semantic-search graphs             # what's connected, and what's indexed

scfh         SCFH         hosted   indexed (9392 records)
brain        hippo        hosted   not indexed
apple        Apple        offline  not indexed

Nicknames, canonical names, per-graph bearer tokens, and the Local API port are read from Roam's own config files, so this project stores no graph configuration of its own:

Each index lives at ~/.cache/roam-semantic-search/<canonical-name>.db — keyed by the canonical name, never the nickname, so the CLI and MCP server always address the same file.

Offline graphs (registry type of offline rather than hosted) cannot be indexed: the /api/<name> Local API path serves hosted graphs only, and rejects them with "Token is valid for offline graph … not hosted". They appear in graphs but build refuses them with that explanation.

Configuration

Graph identity and credentials come from the files above. The remaining environment variables tune everything else, and are all optional:

An explicit flag always wins, then the registry, then the environment. The registry is preferred over GUFFIN_ROAM_API_TOKEN deliberately: that variable names one default graph, and its token is the wrong credential for any other.

CLI

roam-semantic-search graphs                       # connected graphs and their index state
roam-semantic-search build --graph brain          # full fetch → normalize → embed → store (~100 s for ~8k records)
roam-semantic-search refresh --graph brain        # incremental: re-embed only what changed (~2 s when idle)
roam-semantic-search search --graph scfh "why the human must stay responsible" -k 5
roam-semantic-search stats --graph scfh           # store provenance: model, counts, build/refresh moments

--port and --token are needed only to override the registry, or to reach a graph it does not know about.

Unattended refreshes: --require-open-window

Reaching a graph opens it. The Local API is addressed per graph (/api/<name>), and a request naming a graph that Roam Desktop has no window open on makes Roam open one — for an encrypted graph, an unlock prompt that sits there waiting for a human. An hourly scheduled refresh of a graph you keep closed therefore produces an hourly unlock prompt, and the request fails anyway.

--require-open-window gates the run on Roam's window-restore list (user-config.edn, read from disk — the check itself never touches Roam), and skips unless that list names the graph:

roam-semantic-search refresh --graph scfh --require-open-window
# skipped: Roam Desktop has no window open on SCFH

A skip succeeds (exit 0), so a scheduler records no failure and the next run simply retries. An unreadable list skips too — it is not a listing.

What the flag can and cannot do. The check is a necessary condition for reachability, not a sufficient one, so it reduces prompts rather than abolishing them:

• An unlisted graph is certainly unservable, and skipping it is always right.

• A listed graph may still be closed or, for an encrypted graph, locked. The file is a restore list — written when the window set changes, then carried across a relaunch — and it records only that a window exists, never whether the graph behind it is unlocked. A relaunch re-locks an encrypted graph while faithfully restoring its window, so a stale listing sails through the gate and the request prompts anyway.

Nothing observable off the filesystem distinguishes a servable graph from a locked one: the Local API has no graph-agnostic status route, so the request itself is the only test, and issuing it is what raises the prompt. A scheduled refresh of a graph you keep locked will therefore prompt sooner or later — the durable fix is not to schedule one.

The flag is opt-in — an unflagged refresh behaves exactly as it always has — because an interactive refresh of a closed graph is a reasonable thing to ask for, and unlocking the graph is then a deliberate answer rather than an ambush.

A hit shows the Roam uid (usable as a ((ref))), the fused score, each ranking's position (v: vector, k: keyword), the breadcrumb, and the text:

 1. ((9KMmmo5aH))  [block  score 0.0323  v:2 k:2]
    The new Programmer (in the age of AI assistants) › The human Programmer/engineer
    The human also remains the accountability boundary. The assistant can propose; ...

MCP server

roam-semantic-search-mcp serves every connected graph's index over stdio to any MCP client, with four tools:

Staleness is bounded, and never silent. semantic_search answers from the index — a snapshot — but when the last capture is older than the staleness threshold (ROAM_SEMANTIC_SEARCH_MAX_STALENESS, default one hour) it refreshes the index before searching. A failed refresh (most commonly Roam Desktop not running) degrades gracefully to the snapshot. Either way the response's refresh field reports what happened — fresh, refreshed, refresh-failed (with the error), window-closed, or disabled — so a caller who cares about currency checks one field instead of doing timestamp arithmetic.

Set ROAM_SEMANTIC_SEARCH_REQUIRE_OPEN_WINDOW=1 and the automatic pre-search refresh is gated the same way --require-open-window gates the CLI, reporting window-closed and answering from the snapshot. Searching a closed encrypted graph then costs no unlock prompt. An explicit refresh_index call is never gated — asking for it is deliberate, and it is the way to say yes, open the graph, I will unlock it.

Register with Claude Code — no per-graph configuration, since the server reads Roam's registry directly:

claude mcp add --scope user roam-semantic-search -- $(pwd)/.venv/bin/roam-semantic-search-mcp

semantic_search, refresh_index, and index_stats each require a graph argument. That is deliberate rather than an ergonomic oversight: with several graphs connected, a server-wide default would silently answer from whichever graph the process happened to be configured for — a wrong answer that reads exactly like a right one. list_indexes exists so a caller can discover the legal values instead of guessing.

Development

.venv/bin/black .
.venv/bin/ruff check --fix src/ tests/
.venv/bin/pyright          # strict
.venv/bin/pytest

Conventions follow guffin's (Python 3.14, src layout, pyright strict, @validate_call, regex not re, 120-char lines). The index DB contains the graph's text in clear form — treat it like an export, and keep it out of anything synced or shared.