# vault-rag-mcp

[![tests](https://github.com/sevamrk/vault-rag-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/sevamrk/vault-rag-mcp/actions/workflows/ci.yml)

An MCP server that answers questions about a folder of markdown notes. It runs
keyword search and vector search over the same corpus, fuses the two ranked
lists, reranks the shortlist with a cross-encoder, and hands the result to an
MCP client as a tool call.

Everything runs on the machine it's installed on. There's no API key, no
inference endpoint, and no account. The only network access is a one-time model
download from the Hugging Face CDN the first time you index.

> **In 30 seconds**
>
> | | |
> |---|---|
> | **What it does** | Hybrid search over a markdown vault, served to an MCP client. Keyword plus vectors, fused by rank, then reranked |
> | **The hard part** | Two ranked lists with incomparable scores. Fusing on position rather than score is what makes them combinable |
> | **The evidence** | `npm run eval` scores four retrieval configurations over 24 labelled queries |
> | **What it admits** | The cross-encoder reranker loses to plain fusion on the demo corpus, and the evaluation says so |

## The problem

Search over your own notes fails in two distinct ways, and the two aren't
variations of one problem.

Keyword search. BM25 over an FTS5 index. Is exact, fast, and needs no model.
It finds the note that contains the words you typed. Ask it `compaction stall`
and it returns the right note in four milliseconds. Ask it *why did writes get
slow in the middle of the night* and it returns nothing at all, because that
phrasing shares no rare token with anything you wrote.

Vector search has the opposite profile. It embeds the query and finds notes that
sit nearby in the embedding space whether or not the words match, so the
midnight question lands on the compaction postmortem. But exact tokens dissolve.
Error codes, config keys, version strings and surnames come back surrounded by
notes that are merely about the same sort of thing.

The two failure modes are close to independent, which is why running both and
fusing them isn't a compromise between two mediocre answers. A note ranked
highly by both methods is much more likely to be right than one ranked highly by
either. You can watch this on the bundled demo vault:

```
$ npm run search -- "why did writes get slow in the middle of the night"

  keyword (FTS5/BM25)
    (nothing)
  vector (sqlite-vec)
    1. [0.636] knowledge/operations/the-first-question-in-an-incident-is-what-changed.md
    2. [0.599] sessions/orchard/2026-03-05-backpressure-rollout.md
  hybrid (RRF + rerank)
    1. [-8.489] knowledge/operations/the-first-question-in-an-incident-is-what-changed.md
    2. [-9.753] sessions/ferrite/2026-02-03-compaction-stall-postmortem.md
```

## How it fits together

<img src="docs/diagrams/architecture.svg" alt="A vault is chunked on headings into one SQLite file holding both a keyword index and vectors; a query runs both branches, fuses them by rank, reranks the shortlist and applies a recency boost" width="820">

The reranker reorders; it cannot recall. If the right chunk isn't in the
shortlist, nothing downstream will find it. On the demo vault it currently costs
more accuracy than it earns, and the evaluation below says so with numbers.

## The retrieval path

One SQLite file holds all of it: the note rows, an FTS5 index over title and
body, chunk text, and a `sqlite-vec` virtual table of 384-dimensional chunk
embeddings. One file means one lock, one backup, and no second service to keep
in sync with the first.

**Chunking splits on level-two headings**, not on a fixed token window. An
embedding is a summary of whatever text it was given, so a chunk that straddles
a topic boundary produces a vector that sits between two topics and near
neither. Markdown headings are a segmentation the author already wrote down, and
the heading itself is usually the densest line in the chunk, so it gets
prepended to the text before embedding.

**The keyword branch** is FTS5 with the porter tokenizer, ordered by `rank, n.id`.
The secondary key isn't decoration. Bm25 ties are common, `ORDER BY rank` alone
isn't a total order, and a stable sort over an unordered result set means the
top hit is decided by the query plan.

**The vector branch** asks sqlite-vec for the K nearest chunks by L2 distance and
then applies structured filters. It applies them *after* the KNN, because a vec0
table holds only `(chunk_id, emb)` and has no metadata columns to filter on.
That ordering has a sharp edge: filtering an already-truncated top-40 can return
empty for a scope that the corpus genuinely covers, which turns a false negative
into a false positive. When a path scope is present the code escalates K along a
ladder until enough in-scope results survive, the index drains, or the ladder
runs out, and it reports which of the three happened, because "nothing matched"
and "I stopped looking" are different answers. See `src/search/semantic.ts` and
`tests/search-path-scoping.test.ts`.

**Fusion is reciprocal rank fusion**, which combines the two lists by position
and discards the scores. This is the point: a BM25 score is unbounded and
corpus-dependent, a cosine similarity sits in a narrow band, and normalising the
two onto a common scale makes a branch with no good answers look exactly as
confident as a branch with a perfect one. Ranks need no calibration. The cost is
real. RRF cannot express how much better the top hit is than the second.

**The reranker** is a cross-encoder that reads the query and one candidate
together and produces a relevance score. It cannot be precomputed, so it only
ever sees the shortlist: thirty candidates, one batched forward pass, roughly
450 ms on two cores. It's the single most expensive stage and the only one that
scales with the shortlist size. It reorders; it cannot recall. If the right chunk
isn't in the shortlist, no amount of reranking will find it. On the demo vault it
currently costs more accuracy than it earns, for a reason worth reading: see the
evaluation below.

**HyDE is off by default and template-based.** The classical technique asks an
LLM to write a hypothetical answer and searches with the answer's vector. There
is no LLM in this process and adding one would invert the layer relationship, so
what ships instead is a deterministic template expansion that pads the query into
answer-shaped text and fuses the result as a third branch. It costs about 20 ms
warm and helps on vague paraphrased queries.

**Recency is an additive boost applied after reranking**, with a frozen-clock
parameter. The reranker emits signed logits, so a multiplier would flip the sign
on poor matches. Any harness comparing hybrid output has to pass `now` explicitly
or it will drift against itself.

## Running it

```bash
npm install
npm run reindex          # builds data/vault.db from ./vault — about 15s, 33 notes
npm run search           # five demo queries across all three branches
npm run eval             # score four retrieval configurations on 24 labelled queries
npm test
```

The first run downloads two ONNX models (roughly 350 MB total: BGE-small-en-v1.5
for embeddings, ms-marco-MiniLM-L-6-v2 for reranking) into
`node_modules/@huggingface/transformers/.cache`. Every run after that's offline.

To run it as an MCP server:

```bash
npm run build
npm start                # HTTP daemon on :8848, health at /health, MCP at /mcp
VAULT_RAG_TRANSPORT=stdio npm start   # classic one-process-per-client stdio
```

HTTP is the default because the ONNX models are process-global. One daemon means
the models are loaded once for the whole machine instead of once per connected
client, which on a small box is the difference between working and swapping.

Five tools are registered: `vault_search`, `vault_read`, `vault_list`,
`vault_reindex`, and `vault_delta`. The last one is the interesting one. Given a
batch of candidate items it returns which the vault already covers and which are
new, which is what you want before ingesting notes from somewhere else.

## Pointing it at your own notes

```bash
cp .env.example .env
```

Set `VAULT_PATH` to your notes directory and `DB_PATH` to wherever the index
should live, then `npm run reindex`. Nothing else is coupled to the vault's
layout. The indexer reads YAML frontmatter if it's there (`type`, `date`,
`project`, `status`, `source`, `topics`) and works without it; `templates/**` and
`lint-report-*.md` are skipped at scan time; `[[wikilinks]]` are extracted into a
sidecar table and surface as related-note suggestions on the top hit.

The index is a derived artifact. Delete it and rebuild whenever you want.

## The demo vault

`vault/` holds 33 synthetic notes. 132 chunks. Written for this repository.
None of it's anyone's real notes. It describes two fictional systems, a
time-series store called Ferrite and a job queue called Orchard, plus a set of
engineering-practice notes and a handful on baking and running so that topical
scoping and filtering visibly do something.

The topical spread is deliberate. A corpus about one subject cannot demonstrate
that a path scope or a type filter changes the answer, and it cannot produce the
adversarial case in `tests/search-path-scoping.test.ts` where a query's entire
top-40 falls outside the scope being searched.

Also deliberate: three reference pages share a `## Conventions` block verbatim.
Identical text embeds to an identical vector, so those results carry a real
exact-tie group, and the frozen baseline in `tests/fixtures/search-baseline.json`
can therefore detect a tie-ordering regression. A baseline that never sees a tie
cannot.

## What the retrieval actually scores

The claim that fusion beats its parts is checkable here rather than
asserted. One command, about a minute, no API key, and no network once the
models are cached:

```bash
npm run reindex && npm run eval
```

`eval/queries.json` holds 24 queries and, for each, the note or notes that
answer it. Nine are phrased in the corpus's own vocabulary, fifteen the way
someone who had forgotten the vocabulary would phrase them. They were
written by reading all 33 notes, before any configuration was scored, and
none was edited afterwards to change a result. `eval/baseline.json` holds
the last run and is committed, so a regression arrives as a diff; `npm run
eval -- --check` writes nothing and exits non-zero when the numbers move.

recall@k is the fraction of a query's labelled notes inside the top k,
averaged over queries. MRR@10 is the mean of 1/(rank of the first labelled
note). nDCG was left out because the labels are binary and grading them by
eye would invent precision the set doesn't have.

| configuration | recall@1 | recall@3 | recall@5 | MRR@10 |
|---|---|---|---|---|
| keyword only (FTS5/BM25) | 0.271 | 0.313 | 0.313 | 0.375 |
| vector only (sqlite-vec) | 0.625 | 0.813 | 0.938 | 0.823 |
| hybrid, RRF fusion | 0.667 | 0.875 | 0.979 | 0.885 |
| hybrid + cross-encoder rerank | 0.604 | 0.792 | 0.833 | 0.799 |
| ...plus the recency boost (server default) | 0.604 | 0.792 | 0.833 | 0.799 |
| diagnostic: rerank the note, not the snippet | 0.729 | 0.938 | 0.938 | 0.922 |

Split by how the query is worded, MRR@10:

| configuration | own vocabulary | paraphrased |
|---|---|---|
| keyword only (FTS5/BM25) | 1.000 | 0.000 |
| vector only (sqlite-vec) | 0.778 | 0.850 |
| hybrid, RRF fusion | 0.944 | 0.850 |
| hybrid + cross-encoder rerank | 1.000 | 0.679 |
| diagnostic: rerank the note, not the snippet | 1.000 | 0.875 |

Read the keyword row first. It answers every vocabulary query at rank 1 and
returns literally nothing on all fifteen paraphrased ones. 0.000, not a low
score. That's the argument for the second branch, in one line, and it's
why the fusion isn't a compromise between two mediocre retrievers.

Fusion holds up. RRF over the two branches beats both of them on every
column of the first table, and the split table shows what it costs to get
there: 0.944 on the vocabulary half against keyword's perfect 1.000, in
exchange for 0.850 instead of 0.000 on the other fifteen. Losing a little
where one branch is already perfect is the trade the fusion is for.

**The reranker loses on this corpus, and the table says so.** Adding the
cross-encoder to the fused list drops MRR from 0.885 to 0.799, all of it on
paraphrased queries. Five queries where a simpler configuration wins, with
the rank of the first correct note:

| query | keyword | vector | RRF | RRF + rerank |
|---|---|---|---|---|
| clients keep hammering an endpoint that is already refusing them | – | 2 | 2 | 8 |
| is refusing work when the buffer is full a bug or the design | – | 1 | 1 | 4 |
| our search misses notes when the question is worded differently from the note | – | 1 | 1 | 6 |
| how much time does the cross encoder add to a query | – | 1 | 1 | 2 |
| where should a long document be cut up before embedding it | – | 1 | 1 | 7 |

The cause is visible once you print what the cross-encoder is being handed.
It scores `title + snippet`, and the snippet is at most 240 characters of
the best-matching chunk, which for a short note is often the title chunk,
so the passage is little more than the note's own title repeated. The model
then has almost nothing to judge. On the queries in that table the correct
note comes back scored around -11, which is the cross-encoder saying nothing
here looks like an answer, and what you're reading as a ranking is an
ordering over noise. The last row of the table changes exactly one thing,
the passage becomes the note's own text, and MRR goes to 0.922: above fusion
alone, and above every other row. So the reranker isn't the wrong idea
here; the passage it's given is too short.

The fix is deliberately not shipped. It changes what every search returns,
and the evidence for it's 24 queries over 33 synthetic notes.

The recency boost, which is on by default, moves exactly one query on this
set, and downward by one rank. Too small to conclude anything from, which is
itself worth knowing about a default.

### What this does not prove

A 33-note synthetic corpus isn't evidence about a 3,000-note real one. It
shows that the machinery runs, that the two branches fail in the directions
the design claims, and that the numbers reproduce. It doesn't predict the
numbers on a real vault, and the error isn't symmetric: on a corpus this
small almost every note is about something different, which flatters the
vector branch. A vault with fifty notes on one subject is a harder ranking
problem than anything measured here, and it's the ordinary case in a vault
of three thousand.

The labels are binary and mine. On the "bug or the design" query the
reranker's top hit is `reference/orchard-queue.md`, which does state that a
full ingress rejects rather than buffers, a defensible answer that the
label doesn't credit. A larger set with graded labels, written by someone
who didn't build the retriever, would be a better instrument than this one.

## Tests

```bash
npm run reindex && npm test
```

74 assertions across 9 files. They need the index built first, because most of
them query it; the runner says so rather than failing on an assertion.

`npm run eval` is deliberately not one of them. It loads both models and takes
about a minute, and its output is a measurement to read rather than an assertion
to trip. The committed baseline is what turns it into a regression check, on
demand, with `npm run eval -- --check`.

What they cover, and why those things and not others:

| Suite | Guards |
|---|---|
| `search-path-scoping` | the post-KNN filter trap and the escalation that defeats it |
| `search-baseline` | retrieval determinism, tie ordering, and a frozen result set |
| `model-singleflight` | that model loading caches the promise, not the resolved model |
| `concurrency` | the mutex and semaphore the daemon serializes on |
| `http-server` | session lifecycle, shutdown, transport selection |
| `indexer-embed-backfill` | that a note with rows but no vectors is detected and repaired |
| `indexer-malformed-frontmatter` | that one broken note does not abort the reindex |
| `parser-date-normalize` | that YAML `date:` becomes a sortable string, not a `Date` |
| `eval-metrics` | the recall and MRR arithmetic the evaluation table is built from |

Two of these are worth explaining.

`model-singleflight` asserts on the *source text* rather than behaviour, because
exercising the real loaders would download and hold hundreds of megabytes of
weights. The bug it guards is that `if (!model) model = await load()` is
evaluated by every concurrent caller before the first await settles, so N
simultaneous first-queries each start their own load. Caching the in-flight
promise fixes it. A structural assertion is a weaker test than a behavioural one
and it's the one that can run in CI.

`search-baseline` freezes a set of query results and holds retrieval to them, but
only while a corpus fingerprint (note count, chunk count) still matches. Once the
vault changes, the frozen answers can't be asserted and aren't faked, so the suite
falls back to asserting determinism, prints that it did, and stays green. A
frozen assertion against a moving corpus is a nightly false red, and a false red
in the guard layer is how guards get deleted.

It used to assert the scores byte-for-byte too, and that was wrong. On 2026-08-30
it went red in CI on a commit touching a README, a Makefile and a diagram: the
paths and their order were identical and three scores differed at the eighth
decimal place, about 4e-8. That's the ONNX runtime choosing different SIMD kernels
on a different machine, and nothing in this repository can make it reproducible
across machines. So the ranking, meaning which documents come back and in what
order, is asserted exactly, and the scores are asserted to within 1e-6. A ranking
change still fails; hardware noise no longer does. Both directions are covered by
deliberately breaking the fixture: swapping two results fails, moving a score by
1e-4 fails, and moving one by 1e-8 passes.

## Layout

| Path | Does |
|---|---|
| `src/db/` | schema, prepared statements, sqlite-vec loading, the wikilink sidecar |
| `src/indexer/` | scan, parse frontmatter, chunk on headings, embed, extract links |
| `src/search/keyword.ts` | FTS5 / BM25 branch |
| `src/search/semantic.ts` | vector branch, structured filters, the K escalation ladder |
| `src/search/hybrid.ts` | RRF fusion, rerank, recency boost, related-note attachment |
| `src/search/rerank.ts` | cross-encoder, single-flight model load |
| `src/search/hyde.ts` | template query expansion |
| `src/search/order.ts` | the deterministic comparators every sort runs through |
| `src/eval/metrics.ts` | recall@k and reciprocal rank, kept pure and tested |
| `src/tools/` | the five MCP tool definitions |
| `src/http-server.ts` | the shared daemon, one MCP server per session over one database |
| `scripts/search-probe.ts` | the three-branch demo above |
| `scripts/capture-search-baseline.ts` | regenerates the frozen fixture |
| `scripts/eval-retrieval.ts` | the four-configuration comparison above |
| `eval/` | the labelled query set and the committed score baseline |

## Provenance and honesty about origin

This is extracted from a personal notes system that has been running daily since
March 2026 against a private vault of roughly four thousand markdown files. The
retrieval code is that code, with two hardcoded paths replaced by configuration. What didn't come with it: the vault, the nightly extraction pipeline
that fed it, the deployment, and a calibration harness whose labelled evaluation
set scored private material and couldn't be published. `eval/` is the public replacement
for that harness: smaller, over a synthetic corpus, and honest about what it can support.

The numbers left in the comments, a 0.745 novelty threshold, a measured 0.106
overlap between duplicate and novel claims, the language guard that refuses to
score Cyrillic text because the embedding model reports language rather than
meaning for it. Were measured on that private corpus, not on the demo vault. They
are kept because a constant traceable to a measurement is more useful than one
traceable to taste, and they're labelled so nobody mistakes them for something
this repository can reproduce.

## License

MIT.