# Vault MCP

An MCP server and REST API over a local [Obsidian](https://obsidian.md) vault. It
indexes the vault for hybrid search and serves read and write access through two
interfaces backed by one implementation.

## Why

It replaces a plugin with two problems. **Heading targets had to be exact** — the full
ancestor path from the H1 down, which 86% of this vault's notes wrap everything in; here
a bare leaf name works whenever it is unique, and when it is not, the error names the
ancestors to prepend. **There was no retrieval** — finding a note meant knowing its path.
`vault_search` is hybrid, dense vectors and BM25 fused with reciprocal rank fusion,
because this corpus is dense with exact tokens (reg plates, model numbers, policy
references, postcodes) that dense retrieval alone underperforms on.

## Interfaces

**MCP** at `/mcp` — fourteen tools:

| Read | Write |
| --- | --- |
| `vault_search` — hybrid search, `prefix=` to search one folder | `vault_patch` — replace a section |
| `vault_read` — whole note, one `section=`, or a document's extracted text | `vault_replace_text` — substitute exact text, optionally inside one section |
| | `vault_append` — add to the end |
| `vault_list` — browse a folder | `vault_write` — create or overwrite |
| `vault_map` — heading tree as `::` paths | `vault_set_body` — replace the prose, keep the block |
| | `vault_set_frontmatter` — set or delete a key |
| | `vault_delete` — remove a note or a document |
| | `vault_move` — move, rewriting inbound links |
| | `vault_create_folder` — create a folder and its parents |
| | `vault_delete_folder` — remove one, `recursive=` for its contents |

`vault_map` emits `::`-joined paths rather than an indented tree, so the output can be
pasted straight back as a patch target.

**REST** at `/vault/<path>`, mirroring the shape `obsidian-local-rest-api` used — n8n's
HTTP Request nodes speak plain REST and cannot easily build a JSON-RPC envelope, so
migrating a node is a find-and-replace rather than a rewrite.

| Call | Does |
| --- | --- |
| `GET /vault/<path>` | The note's markdown. `?section=` narrows it to one heading. |
| `GET /vault/<path>` with `Accept: application/json` | `{path, content, body, frontmatter}` — `content` is the file byte for byte, `body` the same text with the frontmatter block removed |
| `PUT /vault/<path>` | Create or replace. A document suffix takes the binary branch: bytes stored as sent, answer is JSON. |
| `POST /vault/<path>` | Append, creating the note if absent |
| `PATCH /vault/<path>` | `Target:` a heading, or a frontmatter key with `Target-Type: frontmatter`, the prose with `Target-Type: body`, or literal text with `Target-Type: text`. `Operation: delete` removes a frontmatter key. |
| `DELETE /vault/<path>` | Remove the note |
| `GET /vault/<folder>` | The folder's listing — text, or JSON under `Accept: application/json` |
| `GET /vault/<path>?map` | Frontmatter and heading outline, without the content. What `vault_map` answers, as JSON. |
| `GET /search?q=` | Ranked excerpts as JSON. `&k=` for how many, `&prefix=` for one folder, `&explain` for per-arm ranks and the fusion constants. `POST` works too. |
| `PUT /folder/<path>` | Create a folder and any parent it needs. Already existing is success. |
| `DELETE /folder/<path>` | Remove a folder. `?recursive` to take its contents; without it, a non-empty folder is refused. |
| `GET /frontmatter?key=&value=` | Notes whose field holds that exact value. `&dir=` narrows the walk. Walks the filesystem, never the index — `Workflows/` is excluded from search and is where these notes live. |
| `GET /expiries` | Every dated commitment in the vault, oldest first, with days remaining. `?within=` for a window, `?untracked=1` to list the notes carrying no `expires` field, `?dir=` to narrow the walk. |
| `GET /maintenance` | Runs every checker the vault offers and returns their findings, as JSON and as one markdown block |
| `GET /maintenance/checks` | The same checkers, listed without running them — name, title, description, order |
| `GET /callouts` | Every open callout in the vault as JSON — note, line, type, severity, title, body |
| `GET /healthz` | `ok`, once the port is bound. Depends on nothing, and is the one route with no bearer token. |
| `GET /readyz` | Index, `index.md` and watcher state as JSON. 200 when all three are good, 503 with the same body when any is not. |

A frontmatter `PATCH` takes a **JSON** body, so `"approved"` needs its quotes and `2`
does not — the value is decoded rather than copied. Removing a field is
`Operation: delete` with no body; a `null` value is refused, since no text reads back as
one. `Target-Type: body` replaces the prose and leaves the frontmatter untouched.

`Target-Type: text` is the smallest edit the surface offers, for a table row or a
single field where rewriting the whole section would mean resending every line
around it. It takes a JSON body — `{"find": "...", "replace": "...", "section":
"...", "count": 1}` — rather than headers, because a newline in a header is not a
newline and the text being replaced routinely has one. `find` is literal, never a
pattern. `count` is what the caller *expects*, not a limit: a different number in
the note is refused before anything is written, so a string that turns out not to
be unique cannot be replaced in three places silently.

`/maintenance` and `/callouts` run a stdlib script out of the vault's own `.scripts/`,
because n8n's container has neither Python nor the vault mounted and this one has both.
Neither takes parameters — every argv is a constant, so nothing a caller sends reaches a
command line — and both accept `POST` as well as `GET`. Findings are a 200 however many
there are; only the run failing outright is a 500. What `/maintenance` gets back is
described under [the findings contract](#the-findings-contract), below.

Two deliberate differences from the plugin: the structured read answers
`application/json`, never the vendor `application/vnd.olrapi.note+json` (n8n hands the
vendor type to the workflow as a string, so a node written against the JSON shape would
silently succeed with nothing); and a missing note is `404` where every other rejected
path, target or body is `400`, because "no such note" is the one error a caller routes on.

### Errors carry a code

Every rejected REST call answers JSON with two fields and nothing else:

```json
{"code": "wrong_parent_folder", "error": "a document must be filed directly inside a folder named 'Files', and Home/scan.pdf is not. …"}
```

`code` comes from a closed set declared once in `src/vault.py`. The `400`/`404` split is
too coarse to route on, and the alternative was matching on prose written for a model to
read. Two failures share a code when the caller does the same thing about them —
traversal, a symlink and a write to the generated `index.md` are one `invalid_path` with
the mechanism in the message.

| Code | Status | Means |
| --- | --- | --- |
| `not_found` | 404 | The path is not there. Also an unknown route, and a search `prefix=` matching nothing indexed. |
| `already_exists` | 400 | Pass `overwrite`, or choose another name |
| `not_empty` | 400 | The folder has contents; pass `recursive` |
| `invalid_path` | 400 | Traversal, a symlink, a null byte, a protected path, the vault root |
| `out_of_scope` | 400 | This request's remit is narrower than the caller thinks. Do not retry elsewhere. |
| `unsupported_suffix` | 400 | The wrong kind of file for this verb — including a folder named like a note |
| `is_directory` / `not_a_directory` | 400 | A directory where a file was needed, or the reverse |
| `is_document` | 400 | A document where a note was needed, or a section named on one |
| `kind_change` | 400 | A move may not turn a note into a document |
| `no_such_heading` / `ambiguous_heading` | 400 | The target names none, or several — retry with the full `::` path |
| `no_such_text` / `ambiguous_text` | 400 | The find string is not there, or is there a different number of times than `count` said |
| `no_frontmatter` / `invalid_value` | 400 | Nothing to edit, or not a value this vault's frontmatter carries |
| `unreadable` | 400 | It is there, and the read failed |
| `empty_document` / `wrong_parent_folder` / `extraction_failed` | 400 | No bytes, the wrong folder, or no text inside |
| `document_too_large` | 400 | Over `vault.doc_max_bytes` |
| `invalid_request` | 400 | A missing header, an unparseable body, an unknown verb |
| `method_not_allowed` | 405 | |
| `unauthorized` | 401 | No bearer token. Same body shape as everything else. |
| `not_ready` | 503 | The index is still building — ask again, rather than recording no results |
| `internal_error` | 500 | Only `/maintenance` and `/callouts`, when the checker itself will not run |

This changed the wire format: rejected calls used to answer `text/plain` carrying the
message alone, which is now under `error`. A header would have avoided the break, but
`$json.code` is an expression anyone can write in n8n and a response header is not.
`tests/errors.py` keeps the set closed at both ends.

The MCP surface is unchanged and still answers prose — a model reads the message and
decides what to do next. Both surfaces call `src/operations.py`, so the resolver and the
vault conventions apply once however the caller arrived, and every write bumps the note's
`timestamp` or reports why it could not.

### The surface is a contract now

`openapi.yaml` describes every REST route and is committed rather than generated —
Starlette has no schema to introspect here, since the routes take headers and raw bodies
that no decorator describes.

It exists because the surface is consumed from another repository. While n8n was the only
caller the table above was enough, because a workflow that broke was edited by the person
who broke it; a typed client generated from a document is not, and the failure it prevents
is a field renamed here and still expected there. `code` is the part most worth generating
from: a closed set makes an exhaustive union, and `tests/errors.py` keeps the document,
the source and the wire agreeing in all three directions.

## Expiry dates

`GET /expiries` reads the `expires` frontmatter block that `Meta/Conventions.md` defines
and `.scripts/check_expiries.py` validates:

```yaml
expires:
  - date: 2026-09-25
    what: Annual boiler service due - required to keep the 12-year guarantee valid
```

`/frontmatter` cannot answer this. It matches one key against one exact value, and the
question here is every value before a date — so this is its own route rather than a
parameter on that one.

**One row per date, never one per note.** Conventions is explicit that the dates in a note
must not be collapsed to the soonest, and the reason is worth repeating: once that date
passes and is replaced with a later one, every earlier date it was standing in for becomes
invisible and silently stops being checked. A `min(dates)` here would reintroduce exactly
the bug the convention warns about.

Three groups come back, because they are three different questions. `entries` are live
commitments. `dismissed` are notes carrying `expires: none` *and* a reason — reviewed, and
deliberately not tracked, which is a different state from having no field at all.
`untracked` are notes with no `expires` field, counted by default and listed under
`?untracked=1`: it is most of the vault, and whether a note needs the field is a judgement
nothing here can make — but it is the list that finds the renewal nobody wrote down.

`?within=` filters to a window. Lapsed dates come back whatever it says: a date that has
already passed is the one thing a caller must not have to ask for, and a window that hid
it would make a quiet answer mean two different things.

## The findings contract

Every check in the vault's `.scripts/` emits the same three things: a coloured listing for
a person, one findings document for a machine under `--json`, and — under `--describe` —
the declaration of what it is. The listing and the document are rendered from the same
findings, so they cannot disagree. The contract is written out above `Report` in
`.scripts/vaultlib.py`; this is why it exists.

Before it, each script classified its findings by passing a **colour** to the shared
printer and then hand-wrote its own exit rule at the bottom. Severity was presentational,
so the exit code could not be derived from it, so no two scripts agreed on what a non-zero
exit meant — and this repo carried a sentence per script explaining the disagreement, in a
`nonzero_means` field, so a report would not read 27 hygiene warnings as a clean check.

Promoting severity to a field made the exit code one line of shared code:

| Severity | Means | Exit |
| --- | --- | --- |
| `error` | The vault is broken — a file is corrupt, a guarantee is void, a section is unpatchable, a document is unreachable | forces 1 |
| `warning` | Drift from the standard; the file is intact and nothing is lost | stays 0 |
| `info` | An observation with no action implied | stays 0 |

Exit `2` is new and is the one that mattered most: **the check could not run.** A crashing
checker used to print a traceback and exit 1, indistinguishable from findings. For a cron
job that is survivable; for anything rendering the result it is the difference between
"your vault has a problem" and "your checker is broken", and those must never look the same.

Each finding carries a stable `id`, derived from what identifies it — never from its whole
content, which would make every finding look new on every run, which is how a maintenance
list becomes noise nobody reads. A `fix` key is **reserved and not yet emitted**: when a
finding can describe its own repair it will carry a `kind` matching this surface's
`Target-Type` values, so a finding becomes a button. Declaring it now stops the scripts
each inventing a different shape for it later.

`nonzero_means` is gone, and so is the per-script prose in the markdown block — one
renderer over findings replaced every script writing its own.

**The scripts and this server deploy separately.** The scripts live in the vault mount;
this ships in an image that is rebuilt. They will be out of step, in both directions. So a
script whose stdout does not parse as a findings document is not an error — it is read as
a pre-contract script and carried through as `status: "legacy"` with its text intact. That
is what lets the scripts migrate one at a time, and what stops a rollback of either side
taking `/maintenance` down.

`check_contract.py` validates every other check: declaration and envelope shape, snake_case
codes, a code meaning one thing, the exit code following from the findings, and no two findings
sharing an id. It is itself contract-conforming, so a new script that does not conform fails
the vault's own maintenance run — which is the only enforcement that holds without someone
remembering to look.

### A script is a check if it says so

Discovery is by declaration. A `.py` in `.scripts/` is a check if it answers `--describe`:

```json
{ "schema": 2, "check": "frontmatter", "title": "OKF frontmatter conformance",
  "description": "Validate note frontmatter against the vault's OKF rules.", "order": 30 }
```

Those five fields are also the head of every envelope that check emits, so one run is
self-describing and `check_contract.py` asserts the two agree. `--describe` walks no files
by contract, which is what makes `GET /maintenance/checks` cheap enough for a page to call
on load — and what lets a check that is currently crashing still appear in that list under
its real name.

This replaced two hand-maintained lists that had already drifted: a `NOT_A_CHECK` set in
`check_contract.py`, and a tuple in this repo pairing every script's filename with a title
that was a second, byte-identical copy of the one the script already emitted. Adding a
checker meant editing both. Now it means dropping a conforming script into the vault —
`/maintenance` runs it and a frontend lists it with nothing rebuilt on this side.

On the security of that, plainly, because the note this replaced claimed more than it
delivered: a fixed list stops a *new* file being executed. It never stopped an existing one
being edited, and `check_contract.py` — on that list — has always globbed the folder and run
everything in it, so this server already executed whatever was there, one hop removed. The
control that actually holds is the mount: compose binds `.scripts/` **read-only** into the
container, separately from the read-write vault mount, so nothing reachable from here can
add a script or change one.

The cost of an opt-in rule is that a check with a broken declaration vanishes rather than
failing, which is worse. Two things cover it. `check_contract.py` reports a `check_*.py`
that will not describe itself as an error; and this server still *runs* such a script, as
`legacy`, titled from its filename — the rollback case, where the vault is on schema 1 and
the image is not. Discovery ignores names otherwise, which is why `compare_index_descriptions.py`
needs no renaming.

## Filed documents

`PUT`ting a file with an allowlisted suffix stores the bytes exactly as sent, extracts
the text, and indexes it under the document's own path, so a search result can read
`Home/Utilities/Files/2026-09-17 Kestrel Energy - Contract Confirmation.pdf`.

**A document may only be written directly inside a folder named `Files`.** That is the
containment control: a compromised pipeline token that can carry raw bytes still cannot
drop a file where a note lives. The folder is created on demand, so filing the first
document needs no separate capability.

```
PUT /vault/Home/Utilities/Files/2026-09-17 Kestrel Energy - Contract Confirmation.pdf
→ {"path": "…", "sha256": "…", "size": 80157, "status": "filed",
   "extraction": "extracted", "pages": 1, "has_text_layer": true,
   "extracted_chars": 802, "detail": ""}
```

`status` says what happened to the file and `extraction` what happened to its text —
separate because the interesting case is a scan that files perfectly and extracts to
nothing.

- **The branch is chosen by suffix, never by `Content-Type`.** n8n forwards whatever the
  upstream mail server labelled an attachment; the name the caller chose is the one thing
  it is reliably sure of.
- **Re-sending identical bytes is a satisfied `200`, not a conflict.** The same
  attachment will arrive twice. Different bytes at the same path need `?overwrite=true`.
- **A document is opaque and move-only** — read, filed, renamed, deleted, but not
  patched, appended to or given frontmatter. A move preserves every byte while still
  repointing inbound links, since the note holding the link is what gets rewritten.
- **Extraction is the server's own read**, not the uploading workflow's, so the vault's
  contents do not depend on which workflow delivered the file.
- **A document that yields no text is reported, not smoothed over** — `extraction` says
  `needs_ocr` or `no_text` and `extracted_chars` is `0`. The vault's
  `check_documents.py` turns that into a finding, alongside a document in `Files/` that
  no note links to.

PDF text is recovered with PyMuPDF4LLM — inferred headings, tables as markdown, page
boundaries as the structural unit where a document has no headings of its own. Scans are
OCR'd through Tesseract where installed; `vault.doc_ocr: false` turns it off. Uploading is
REST-only, since a 10 MB PDF would be 13 MB of base64 emitted a token at a time.

## How a search result is chosen

Fusing the two arms is not the last step. Four rules decide what a caller actually sees,
each of them measured on 73 real queries with [the relevance
harness](#retrieval-relevance):

- **Fused scores are normalised to `(0, 1]` internally, and are not returned.** A raw
  weighted RRF sum is comparable with nothing, so each is divided by the largest
  available - which MMR and the document weight both need, since they have to share one
  scale. It is not returned, because it reads as a confidence and measures as close to
  the opposite of one: see [what a result carries](#what-a-result-carries).
- **A filed document ranks below the note curated from it** — every document chunk's
  fused score is multiplied by `0.7`. A note is written *from* its source, so when both
  answer they answer the same question, and the document does it at the length that wins
  on embedding similarity. This was `0.9` while documents were a small minority of the
  index; at 27% of chunks, and with single PDFs running to 85 chunks against a one-chunk
  note, a tenth off each score no longer offsets the number of attempts a long document
  gets at the candidate window. `0.7` halves the slots documents take without costing a
  single answer its rank. The rule is flat and vault-wide, which is what makes it work
  for documents nobody has filed a note beside yet.
- **No note may hold more than a third of the results.** A note that chunks six ways could
  take every slot at the default `k`; results are picked by maximal marginal relevance
  under a cap of `ceil(k/3)`. It reorders rather than truncates, so a query whose only
  answers live in one note still gets `k` of them. The cap is load-bearing — removed
  altogether, recall@6 falls from `0.9863` to `0.9726` — and it carries more of the work
  than it used to, because `MMR_LAMBDA` was raised to `0.9` once the diversity penalty was
  measured to be hiding answers that live in a note's second chunk.
- **A query that names one thing is answered by that thing.** When the rarest of a
  query's terms appears in at most five chunks *and* one chunk holds every term, BM25's
  top hit is pinned to rank one and exempted from diversification. Asking about the
  rarest term rather than all of them is what lets a hyphenated part number match when
  one piece is common; requiring one chunk to hold every term stops a question that
  merely contains an unusual word pinning whichever note uses it. It pins one chunk per
  *file*, not one chunk — an account number lives in the note and in the statement filed
  beside it, and both are answers.

### What a result carries

Each result names **which arm found it** - `both`, `dense` or `lexical` - and whether it
was `pinned` by the identifier override. It does not carry a score, and that is a
deliberate removal rather than an omission.

The fused score was measured against 12 questions this vault cannot answer at all, beside
the 73 it can. The unanswerable ones scored *higher*: a median top score of `0.966`
against `0.876` for the correct hits of the answerable set, and `"what is the excess on
my caravan insurance"` - there is no caravan, and no note mentions one - came back at a
flat `1.000`. Raw cosine and raw BM25 separate the two sets no better.

The cause is what RRF is. It fuses **ranks**, never magnitudes, so the similarity that
would say how good a match is has already been thrown away before there is a score to
report. Both arms rank every chunk, so a query about nothing still has a rank 1 in each,
and when they agree it scores 1.0. The same arithmetic puts every exact identifier match
at `0.3333` - the most precise retrieval this stack does, scoring lowest on its own
number - because only BM25 can see an identifier.

So the scale never meant quality. It meant arm agreement, which is worth knowing and is
now said in a word instead. `lexical` is the one to act on: BM25 matched the query's
words and the embedder did not agree the chunk is about them. On an identifier that is
correct. On a question phrased in prose, read the excerpt before believing it.

`lexical` also arrives almost only beside `pinned` — **32 of the 37 lexical hits** across
the 73-query set — and the reason is arithmetic rather than habit. A chunk the embedder
ranked *fiftieth* and BM25 never saw fuses to `0.3697`; one BM25 ranked *first* and the
embedder never saw fuses to `0.3333`. Once the dense window is full — any corpus of 50
chunks or more — a lexical-only chunk is below every dense-only chunk whatever either arm
actually thought, so the identifier override is the only door it has. That is why the
override exists, and it is why weakening it strands exact matches rather than merely
deranking them.

Across the same set the labels fall out as 45% `dense`, 43% `both`, 10% pinned and 1%
lexical-unpinned, so `both` and `dense` are what a caller mostly sees.

Results are also not in score order - diversity, the per-note cap and the override all
reorder after scoring - which is one more thing a returned number invited a reader to
get wrong.

#### `?explain`

An opt-in `?explain` (on `GET` or `POST`, any of the truthy spellings the other flags
take) adds two things and changes no ranking. Every result gains an `explain` object:

| field | meaning |
| --- | --- |
| `dense_rank` | 1-based position in the embedder's candidate window, `null` if that arm never returned the chunk |
| `lexical_rank` | the same for BM25 |
| `fused` | the normalised RRF score, **before** the document weight |
| `weight` | the demotion multiplier - the document constant for a filed document's chunk, `1.0` for a note |
| `score` | `fused * weight`, the number the result was actually ranked on |

The response also carries a top-level `explain` with the constants that produced those
numbers - `rrf_k`, `sparse_weight`, `candidates`, `document_weight` - so a caller can
recompute `fused` from the two ranks rather than trust it. This is the same diagnostic
`matched` gives in a word, in arithmetic instead: it exists so a caller can inspect *why*
one result led another, and `fused` is still not a confidence. It measures arm agreement,
and on this vault an unanswerable query scores it higher than an answerable one - which is
why it stays off the default response and off the MCP tool.

### Searching one folder

`prefix=` narrows the corpus to one folder **before** the candidate window, not after,
and on this vault that is the difference between working and not: fusion considers 50
candidates per arm over the whole corpus, and `Home/` is 4.4% of chunks — filtering
afterwards would leave about two candidates for a question the folder answers perfectly.

A prefix matches case-insensitively against whole path segments, so `home/mortgage` finds
`Home/Mortgage` and `Home/Car` does not take `Home/Cards.md`. One matching nothing
indexed is a `404` listing the folders that *are* indexed — "your filter names nothing"
and "the vault does not answer this" are different facts. A folder in
`SEARCH_EXCLUDE_DIRS` cannot be searched under at all.

The rarity test behind the lookup override still asks about the whole vault: a reg plate
is no less of an identifier for being looked for in one folder. Only the chunks it can
pin are narrowed. Under a small prefix candidates run out, so the per-note cap's backfill
branch fires routinely — those tail results are correct but their scores do not descend.

Two things feed retrieval that are worth knowing about:

- **The lexical arm stems.** `tokenize()` lowercases, splits on anything outside
  `[a-z0-9]`, drops stop words and runs Porter2, so "readings" finds "reading" and
  "renewing" finds "renewal". `nomic-embed-text` is still three tokens, and the stemmer
  leaves identifiers alone.
- **A section that is one flat list chunks per item**, so a log or an inbox does not
  become one embedding averaging twelve subjects. The rule asks for seven items, one list
  rather than two, bullets rather than numbers, a median item of at least eight tokens,
  and something left after stripping links — which keeps a recipe method, a list of film
  titles, `## Related notes` and an `## Account` block whole.

## Folders

- **A folder may not be named like a file.** Every component is checked:
  `Notes/Ideas.md/Drafts` would leave a directory called `Ideas.md` that the chunker,
  the link rewriter and the index would all read as a note.
- **Creating one that exists is success**, the same rule as re-uploading identical bytes.
  A *file* at that path is a real collision and is refused.
- **The confirmation names every folder it made**, not only the leaf, because
  `parents=True` turns one mistyped path into four directories.
- **Deleting is empty-only unless asked twice.** A non-empty folder is refused and the
  refusal lists what is inside; `?recursive` is the caller's only other move, and not one
  to make blind.
- **A recursive delete refuses to cross anything protected.** A dotted entry or symlink
  anywhere in the tree stops the whole delete and names the blocker; nothing is removed.
  A partial delete is the worst outcome, since the caller is told it worked. Documents
  are deleted with the rest and counted, so the caller knows to check the `## Documents`
  rows that pointed at them.

**It is a separate route from `/vault/<path>` deliberately** — a recursive delete is the
most destructive verb here and there is no trash, so a `DELETE` to the path n8n already
calls for notes cannot remove a tree however the path is spelled. The vault root is
refused explicitly: `Path("").parts` is empty, so the protected-path test has no
component to match on and answers *false* for the root. Without that check,
`DELETE /folder/` is `rmtree` on the whole vault.

## The index is generated

The vault's root `index.md` is one line per note — title, link, `description` — under
headings mirroring the folder tree. Nothing in it is a judgement call, so
`src/indexdoc.py` derives it rather than asking a model to remember to update it; the
rationale and every fallback are in that module's docstring.

It is rebuilt from the **filesystem watcher**, not the write path, so an MCP call, a REST
`PUT` and someone typing in Obsidian all reach it the same way. A full scan runs at
startup; each change after that re-reads a single note. It only writes when the rendered
body differs, so an edit changing nothing it displays leaves `index.md` and the git
history alone. `index.md` is protected from every writer — a tool accepting a write to it
would teach the caller the edit worked. Fix a wrong line by fixing the note's `title` or
`description`.

Generated note series — the folders in `INDEX_EXCLUDE_DIRS` — get one line in their
parent section rather than a line per note, and only when the folder exists.

## Restarts, and what one costs

Two routes answering two questions that must not be conflated. An unhealthy container is
one something restarts, and restarting to recover a single broken subsystem takes out
every working one with it — so **`/healthz` depends on nothing**. It returns `ok` once
the port is bound, reads no state, and is what the image's `HEALTHCHECK` calls. It is
also the one route with no bearer token, so the healthcheck command does not carry the
key and `docker inspect` does not show it.

`/readyz` is the diagnosis and is never acted on automatically. It reports the index
(built, building or failed, with counts and last build time), `index.md`, the watcher and
the cache — **the same body shape on 200 and 503**, so a failure is read from the fields
a success was read from. A vault whose watch thread has died still answers every read
correctly from a slightly older index: worth reporting, not worth a restart. It also
names the running build, which is what AGPL section 13 asks of a service with no UI.

### The index cache

A restart used to re-read and re-embed the whole vault. It now reuses whatever has not
changed, keyed by content, in a single file outside the vault. On the real vault (208
files, 2223 chunks) a cold start is **40 s**, a warm one **7.8 s**, and one after a
chunker change **17.3 s** — measured over a network mount, where the walk alone is 2.4 s;
the same vault warms in 3.3 s on the server.

Two stores rather than one, and that is the whole design:

    chunks    (vault path, sha256 of the file's bytes)  ->  the chunks it made
    vectors   sha256 of a chunk's embed_text            ->  its embedding

Each invalidation then costs what it should. A **chunker or extractor change** drops the
chunks and keeps every vector, since an embedding is a function of the text alone. A
**model change** drops the vectors and keeps the chunks. A **note edit** drops one file's
chunks and only the vectors whose text moved. A **rename** drops the chunks, since the
path is in them.

The chunk store's key includes a digest of the *source* of the modules that produce
chunks, so a chunker change invalidates it without anybody remembering to say so.
Editing a comment invalidates it too — ten seconds once, against serving stale chunks
until somebody notices search has gone strange.

It is an optimisation and never more: a warm build produces the same chunks in the same
order and the same matrix row for row, asserted in `tests/cache.py` and confirmed
bit-for-bit against `nomic-embed-text` on the real vault. A corrupt, unreadable or
unwritable cache costs a slower start and nothing else. It holds the vault's text outside
the vault, so the directory is `0700` and the file `0600`; a blank `index.cache_path`,
or `INDEX_CACHE_PATH=` empty, turns it off.

What it does not cache is tokenising, which is 5.3 of a warm start's 7.8 seconds. That
used to be repeated for the whole corpus every time one note was saved; the index now
carries its token lists forward, and a note edit costs **133 ms**.

## Scoped writes

`/mcp/only/<path>` is the same MCP surface with this request's writes confined to one
note or one folder. Reads are never scoped — an agent confined to one note still has to
read the conventions and whatever that note refers to. `vault_move` is refused outright
while a scope is set, because rewriting inbound links touches every note pointing at the
source.

It rides on the URL rather than a header because that is the part a caller can vary per
call: n8n's MCP Client node takes auth from a static credential but its endpoint from an
expression, so one agent with one tool list can be handed a different remit per
invocation. The scope cannot outlive its request — the transport is stateless and a
`ContextVar` keeps concurrent requests apart.

This exists because an agent told in prose to "carry nothing out" replaced a section of
the vault's root `index.md` while revising an unrelated note. A sentence in a prompt is
not a guard.

## Configuration

Configuration is a YAML file, with the environment able to override any of it:

```
built-in defaults  <  config.yaml  <  environment
```

Every layer is optional. With no file and no variables the defaults run, which is why an
existing deployment that adds neither behaves exactly as it did before the file existed.
Copy [`config.example.yaml`](config.example.yaml) — it is the whole surface, commented,
every value at its default — and delete everything you are not changing rather than
carrying a copy of the defaults that will drift.

`config.yaml` is looked for beside the package: `/app/config.yaml` in the image, the
repository root otherwise. `VAULT_MCP_CONFIG` names a different one and *requires* it to
be there, because asking for a file by name and silently getting defaults from a typo is
the failure this is meant to stop. Set empty, it skips the search entirely.

Keys are checked against the schema at startup, so a misspelling refuses to boot instead
of quietly taking the default for the rest of the deployment's life.

**`VAULT_MCP_API_KEY` is required, and is the one setting with no file form.** The server
refuses to start without it rather than treating an empty key as "auth off", and it is
read from the environment only — a config file gets committed sooner or later, and this
one guards finances, insurance and addresses as plain text. A key written into the file
anyway is a startup error, not a setting ignored.

| Key | Variable | Default | Purpose |
| --- | --- | --- | --- |
| — | `VAULT_MCP_API_KEY` | — | Bearer token. **Required**, environment only. |
| — | `VAULT_MCP_CONFIG` | `config.yaml` beside the package | The config file itself. Must exist if set; empty means none. |
| `vault.path` | `VAULT_PATH` | `/vault` | Vault root inside the container |
| `vault.doc_suffixes` | `DOC_SUFFIXES` | `.pdf` | Binary document types the vault will carry. Each needs an extractor, so adding one is a code change. |
| `vault.doc_files_dir` | `DOC_FILES_DIR` | `Files` | The folder name a document upload must land directly inside |
| `vault.doc_ocr` | `DOC_OCR` | `true` | OCR a document with no text layer, where Tesseract is installed |
| `vault.doc_max_bytes` | `DOC_MAX_BYTES` | `52428800` (50 MiB) | Ceiling on one uploaded document. `0` disables the check. |
| `server.host` / `server.port` | `BIND_HOST` / `BIND_PORT` | `0.0.0.0` / `8080` | Listen address |
| `server.allowed_hosts` | `MCP_ALLOWED_HOSTS` | `vault-mcp:8080`, `127.0.0.1:8090` | Host-header allowlist |
| `embedding.ollama_url` | `OLLAMA_URL` | `http://ollama:11434/v1` | OpenAI-compatible embedding endpoint |
| `embedding.model` | `EMBED_MODEL` | `nomic-embed-text` | Embedding model |
| `embedding.dims` | `EMBED_DIMS` | `768` | Embedding dimensions |
| `embedding.batch_size` | `EMBED_BATCH_SIZE` | `64` | Embedding requests per batch |
| `embedding.max_attempts` | `EMBED_MAX_ATTEMPTS` | `5` | Attempts per embedding batch before giving up |
| `embedding.backoff_seconds` | `EMBED_BACKOFF_SECONDS` | `1.0` | First wait between attempts; it doubles from there |
| `embedding.backoff_max_seconds` | `EMBED_BACKOFF_MAX_SECONDS` | `30.0` | Ceiling on that doubling |
| `chunking.target_tokens` | `CHUNK_TARGET_TOKENS` | `400` | Target chunk size |
| `chunking.overlap_tokens` | `CHUNK_OVERLAP_TOKENS` | `60` | Overlap between chunks |
| `chunking.min_tokens` | `CHUNK_MIN_TOKENS` | `120` | Below this, a chunk merges into its neighbour |
| `search.default_k` | `SEARCH_DEFAULT_K` | `6` | Default result count |
| `search.exclude_dirs` | `SEARCH_EXCLUDE_DIRS` | `Workflows`, `Reports`, `.obsidian` | Folder *names*, left out of the search index |
| `index.exclude_dirs` | `INDEX_EXCLUDE_DIRS` | the six generated series | Folder *paths*, left out of `index.md` |
| `index.build_max_attempts` | `INDEX_BUILD_MAX_ATTEMPTS` | `5` | Attempts at the initial index build before giving up. `1` disables retrying. |
| `index.build_retry_seconds` | `INDEX_BUILD_RETRY_SECONDS` | `5.0` | First wait between those attempts; it doubles from there |
| `index.cache_path` | `INDEX_CACHE_PATH` | `/cache/index.npz` in the image, `$XDG_CACHE_HOME/vault-mcp/index.npz` otherwise | Chunk-and-vector cache. Empty disables it. |
| `index.cache_flush_seconds` | `INDEX_CACHE_FLUSH_SECONDS` | `60.0` | How often an edited cache is written back. `0` writes at build and shutdown only. |
| `index.reconcile_seconds` | `INDEX_RECONCILE_SECONDS` | `900.0` | Full reconciliation pass over `index.md`. `0` disables it. |
| `watch.debounce_seconds` | `WATCH_DEBOUNCE_SECONDS` | `2.0` | Filesystem-watch debounce before reindexing |

Lists take either spelling. A YAML sequence is the natural one in a file; a
comma-separated string is all a variable can say, and is accepted in both places so the
same setting does not mean two things in two layers.

```yaml
search:
  exclude_dirs: [Workflows, Reports, .obsidian]   # or: Workflows,Reports,.obsidian
```

Quote anything containing a colon — an unquoted `vault-mcp:8080` is a mapping to YAML,
not a string.

`index.cache_path` has three states rather than two, in both layers. Omit the key and it
takes the default; write the key with nothing after the colon — YAML's null, or an empty
`INDEX_CACHE_PATH=` — and the cache is off, which is how you say "do not write my vault's
text anywhere but the vault".

Both exclusion lists take **root-relative paths**, matched as a prefix from the vault
root, case-insensitively, ending on a folder boundary:

```yaml
search:
  exclude_dirs: [Workflows, Reports, .obsidian]       # whole trees
index:
  exclude_dirs: [Workflows/Approvals, Reports/Vault Maintenance]   # the series only
```

So `Workflows` takes everything under the top-level folder, `Workflows/Approvals` takes
one series, and `Approvals` alone matches nothing in either list. A partial segment is not
a prefix — `Workflow` does not take `Workflows/`, and `Workflows` does not take
`Workflows Archive/`. Dotted directories are excluded at any depth whatever these say.

They still hold different things, and that is the decision rather than an oversight:
search may drop `Workflows/` and `Reports/` wholesale, and `index.md` must not, because
curated notes live inside both. What they no longer differ in is how they match. They did
until 2026-09-20 — search took bare folder names against any part of a path — and because
the names are parallel, a list of paths written into the search one matched nothing and
excluded nothing, without an error.

`index.exclude_dirs` mirrors the "Excluded folders" table in the vault's
`Meta/Conventions.md`; that table, this list, `.scripts/check_frontmatter.py` and the
vault's `.gitignore` are four copies of one list and have to move together.

The search index is built at startup and kept current by the watcher, which feeds
`index.md` independently: search needs Ollama and can be slow or unavailable, while the
navigation document needs neither and must not stop updating because an embedding
endpoint is down.

## Running

CI builds the image from each commit on `main` and publishes it as
`ghcr.io/aidan-kay/vault-mcp`, tagged `latest` and with the commit SHA. To build one
yourself, from a checkout - the image copies the working tree, so this builds what is on
disk, uncommitted changes included:

```bash
docker build -t vault-mcp .
docker run --rm \
  -e VAULT_MCP_API_KEY=<token> \
  -v /path/to/vault:/vault \
  -p 8080:8080 \
  vault-mcp
```

That is the whole of it on defaults. To change any of them, mount a config file over the
one place the server looks:

```bash
docker run --rm \
  -e VAULT_MCP_API_KEY=<token> \
  -v /path/to/vault:/vault \
  -v ./config.yaml:/app/config.yaml:ro \
  -p 8080:8080 \
  vault-mcp
```

The image ships `config.example.yaml` rather than a `config.yaml`, so a container with
nothing mounted there runs on defaults and whatever `-e` you pass. `-e` still wins over
the file, which is what keeps one deployment's port a flag rather than a rebuild.

The image carries a `HEALTHCHECK` against `/healthz` and writes its index cache to
`/cache`. Mount a volume there to keep it across `up --force-recreate`. A *named* volume
inherits the image's ownership and works as it is; a bind mount arrives owned by root and
needs chowning to uid 1000, and an unwritable cache is logged once and then costs only a
slower start.

## Security

- **Bearer auth on both surfaces**, failing closed on an unset key.
- **Path containment** in `safe_resolve()` — the single control on where writes land,
  since the vault is mounted read-write. Encoded traversal, `.git`, `index.md` and
  non-`.md` writes are all rejected.
- **Symlinks are refused outright**, checked on the *unresolved* path so `resolve()`
  cannot follow one first. They are creatable on this mount, so this fails loudly rather
  than reasoning about the window between resolving a path and replacing a file.
- **Host-header allowlist**, so the MCP transport is not reachable by DNS rebinding.
- **Per-request write scoping** on `/mcp/only/<path>`, above.
- **One unauthenticated route**, `/healthz`, matched by equality rather than prefix so
  nothing merely starting with it is exempt. It reads no state and returns a constant.
- **The index cache holds the vault's text outside the vault**, the one place this server
  puts it — `0700` and `0600`, or a blank `index.cache_path`.
- **The recursive delete is on its own route**, refusing dotted entries, symlinks and the
  vault root.

## Tests

```bash
python -m tests.run                 # everything, against the committed fixture vault
python -m tests.run --real-vault    # the vault readers against VAULT_PATH instead
python -m tests.run rest indexdoc   # just these
```

Each script also runs on its own, which is what you want when one fails: `primitives`,
`resolve_all`, `resolve_leaves`, `write_scope`, `documents`, `chunker`, `retrieval`,
`indexdoc`, `rest`, `cache`, `embedder`, `errors`, `folders`, `parity`, `config`, and
`relevance.eval`.

Three answer questions about the surface rather than about one function. **`parity`**
runs one case per MCP tool through both surfaces against a freshly rebuilt vault and
compares the bytes left behind: the two may *say* different things and not *do* different
things. It asserts the five deliberate asymmetries, and a tool added without a case here
fails it. **`errors`** keeps the error-code set closed at both ends, reading `src/` with
`ast` and failing on a raise site naming no code, a code that is not declared, and a
declared code that nothing produces. **`folders`** asserts every refusal a recursive
delete makes twice: that it refused, and that the tree is still there afterwards.
**`config`** asserts the precedence above — defaults, then file, then environment — by
reloading `src.config` over a written file per case, since nothing downstream can tell
which layer a value came from. It also loads `config.example.yaml` for real and checks it
still matches the code's defaults, so a renamed setting fails the suite rather than a
deployment.

`tests.run` gives each script its own subprocess because `src.config` resolves settings at
import and `tests.indexdoc` points `VAULT_PATH` at a temp tree before importing `src`, so
two scripts wanting two different vaults cannot share an interpreter.

Most scripts build their own temp vault; `chunker`, `retrieval` and `embedder` need none.
`primitives`, `resolve_all` and `resolve_leaves` read a vault and assert against what is
in it — [`tests/fixtures/vault`](tests/fixtures/vault) unless you pass `--real-vault`.
The fixture reproduces this vault's *shapes* rather than its contents: H1-wrapped and
unwrapped notes, duplicate leaf headings, a flat-list note, a note long enough to chunk
six ways, identifier-dense notes, generated series that must stay out of the index, and
three PDFs under `Files/` — two with a text layer and one deliberately without.
`primitives` asserts POSIX file modes and symlink refusal, so three of its checks fail on
Windows for want of privileges; it passes on Linux and in CI.

### Retrieval relevance

```bash
python -m tests.relevance.eval                       # fixture, offline, deterministic
python -m tests.relevance.eval --update-baseline     # record a deliberate change
python -m tests.relevance.eval --vault /media/Share/Vault \
    --queries tests/relevance/private.json --embedder ollama
```

Recall@k, MRR and per-note concentration over a committed query set against a committed
baseline: a query that hit at rank 3 may not start missing, and an improvement is
reported rather than failed. This is where the measured claims above come from, and the
gate for any later change to `SPARSE_WEIGHT`, `LOOKUP_MAX_MATCHES`, the tokeniser or the
chunker.

It runs without Ollama by hashing tokens into `EMBED_DIMS` buckets for the dense arm —
deterministic everywhere, and honest about the cost: the stub has no semantics, so
queries needing them are tagged `dense` and reported without being scored. The fixture
run measures the lexical arm, the tokeniser, the fusion, the lookup override, document
retrieval and single-source concentration. Four of its queries can only be answered by a
filed PDF, because "documents are indexed" is a claim that needs a query which fails when
indexing them stops. The real vault and real embedder are a local run against a query set
that stays out of git, since the queries name real accounts — see
[`tests/relevance/private.example.json`](tests/relevance/private.example.json).

`tests.chunker` and `tests.retrieval` are the unit half: the relevance suite says whether
retrieval got better, these say why. They carry the cases the fixture corpus cannot reach
— 27 notes never exhaust their candidates, so the cap's backfill branch never runs there.
Both were checked against six mutations of the code they cover; each one failed them.

## Licence

AGPL-3.0-or-later. See [LICENSE](LICENSE). Copyright (C) 2026 Aidan Kay.

The copyleft is deliberate rather than inherited: PDF extraction uses
[PyMuPDF](https://pymupdf.readthedocs.io/), which Artifex dual-licenses under AGPL-3.0 or
a commercial licence, and taking the AGPL half means this project takes it too.

The practical consequence for anyone running it: if you let other people interact with
your instance over a network, AGPL section 13 obliges you to offer them the source of the
version you are running. No image is published anywhere, so every deployment is someone's
own build from this repo.