# Document Index MCP

Indexes documents on your computer so AI agents can retrieve only the relevant, source-located
passages.

Point it at a folder of PDFs, Word files and Markdown. It builds one SQLite index on your machine.
An agent can then search that library and read short, bounded passages — each carrying the page or
section it came from, so a quotation can be checked against the original.

Uploading a folder of course PDFs into an AI chat is slow, unreliable and expensive in context. This
retrieves locally instead: embedding runs on your CPU, search returns snippets rather than documents,
and a body read is hard-capped.

**Status: beta.** Four formats, all load-bearing: Markdown, plain text, PDF — including scanned PDFs,
via automatic in-process OCR — and Word.

EPUB and PowerPoint readers existed and were **removed** in August 2026 rather than finished. Both
could cite confidently and wrongly — an EPUB locator named a spine file while calling it a chapter, a
chart-built deck indexed its titles and none of its data — and neither had read a file in real use. A
format that can mislead is worse than one that is absent. [docs/roadmap.md](docs/roadmap.md) has the
full reasoning.

For a slide deck, convert it first and ingest both outputs — a PDF of the slides, one page each, and a
Markdown file of the speaker notes:

```bash
pwsh scripts/convert-for-ingest.ps1 "C:\Users\you\Library\Lectures" -Recurse
```

Both matter. A PDF export drops speaker notes entirely, and on a real 28-slide deck 22 slides carried
notes holding figures that appear nowhere in the slide text — the slides were artwork.

⚠️ **That script is Windows-only.** It drives your installed Word and PowerPoint through COM, so it
needs Microsoft Office. On macOS and Linux a deck has no route in at all: convert it to PDF by
whatever means you already have and ingest that, knowing the notes are lost.

---

## Install

Requires **Node 22 or newer** (developed on 24), on **Windows x64, Linux x64 or macOS**. That limit is
inherited rather than chosen — the embedding model's tokenizer ships binaries for exactly those
targets. On Linux arm64, Alpine or Windows-on-ARM the install succeeds and the first search then fails
from inside a dependency, which is a miserable way to find out.

```bash
git clone https://github.com/ekelly95/document-index-mcp.git
cd document-index-mcp
corepack enable   # provides the pnpm 11 that package.json pins
pnpm install
pnpm build
```

Then choose a **library root**: one folder holding the documents you want indexed. It is a security
boundary as well as a convenience — the server refuses to read anything outside it, including through
a symlink that lexically passes but physically escapes.

Pick it before you ingest anything. `source_path` is stored relative to the root. Moving the whole
library folder, index included, needs nothing. Pointing an existing index at a *different* folder —
the parent, a subfolder — is flagged at startup and in the library listing, and
`pnpm rebase --library=<new root> --db=<index>` re-expresses every path against it, or refuses and
changes nothing if a document would fall outside.

The index is a snapshot of each file as it was ingested, but it knows when that snapshot is stale.
Every search hit, every passage read and every library entry carries a `source_status`: `present`,
`changed` (edited since indexing, so a quote may be out of date) or `missing`. It is a stat
comparison, never a re-read. To catch up:

```bash
pnpm sync --library=/path/to/your/library "Papers" --recursive
```

That re-ingests changed files (keeping their titles), retries failed ones whose file has since
changed, ingests new files under the named folders, and lists missing ones. It never deletes:
a missing file looks the same as an unmounted drive or a sync client mid-download, so dropping one
is `delete_document`'s job. `--dry-run` shows the plan without changing anything.

### Register with Claude Desktop

Merge this into `claude_desktop_config.json` — never overwrite it, the file also holds your
preferences. It lives at `%APPDATA%\Claude\claude_desktop_config.json` on Windows, and
`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS.

```json
{
  "mcpServers": {
    "document-index": {
      "command": "/absolute/path/to/node",
      "args": ["/absolute/path/to/document-index-mcp/dist/index.js"],
      "env": {
        "DOCUMENT_INDEX_LIBRARY_PATH": "/absolute/path/to/your/library"
      }
    }
  }
}
```

Use the **absolute** path to the Node binary: a GUI-launched application does not reliably inherit
your shell PATH. Then quit Claude Desktop completely and reopen it — on Windows it persists in the
system tray, so closing the window is not enough.

### Register with Codex

```toml
[mcp_servers.document-index]
command = '/absolute/path/to/node'
args = ['/absolute/path/to/document-index-mcp/dist/index.js']
startup_timeout_sec = 30

[mcp_servers.document-index.env]
DOCUMENT_INDEX_LIBRARY_PATH = '/absolute/path/to/your/library'
```

### First run

The first ingest downloads the embedding model (`bge-small-en-v1.5`, ~65 MB) into
`<library>/.document-index/models`, once, and checks it against a pinned SHA-256. Searching is
entirely local. [Privacy](#privacy) has the whole network story — it is two downloads and nothing
else.

If the library sits in a synced folder (OneDrive, Dropbox, iCloud), put the index and model cache
outside it with `DOCUMENT_INDEX_DB_PATH` and `DOCUMENT_INDEX_MODEL_CACHE` (or `--db=` / `--models=`).
A sync client uploading SQLite's `-wal` and `-shm` files mid-write is a conflict waiting to happen,
and nobody needs the model in their cloud storage.

For a whole library at once, use the bulk CLI rather than ingesting file by file in chat:

```bash
pnpm ingest --library=/path/to/your/library "Papers" --recursive
```

⚠️ **Name a subdirectory rather than `.` unless you are certain what is under the root.**
`--recursive` walks the entire tree, skipping only dot-directories, so a root with an application-data
directory beneath it turns one command into a multi-hour sweep that fills the index with junk. This is
the main argument for a dedicated library folder rather than your home directory.

After upgrading, or whenever the index format changes, rebuild it. The server refuses an index built
by an older schema and says so; rebuilding keeps every document's title and `document_id`, and the
`chunk_id` of every passage whose text came out the same:

```bash
pnpm reindex --library=/path/to/your/library
```

A document is read whole into memory, so files above 512 MB are refused rather than attempted. Raise
it with `--max-file-mb=` or `DOCUMENT_INDEX_MAX_FILE_MB`; the refusal names both the file's size and
the limit, so you know which to change.

---

## Formats

| Format | Locators | Structure comes from | Known limits |
|---|---|---|---|
| `.md` | `section` (`sec-N`, advancing at each H1/H2) | ATX headings | Block text is sliced from the source, never re-serialized |
| `.txt` | `section` | Setext underlines, numbered sections, ALL-CAPS lines, named divisions | A flat outline may be correct rather than a failure |
| `.pdf` | `page`, plus `printed_label` where the printed number differs | Embedded bookmarks (named destinations included) refined by heading styles (size + font). A bookmark that points partway down a page starts its section there; one that names only the page starts it at the top | Tables are read as prose; running headers, page numbers and citation markers are dropped as furniture; scans and mojibake go to OCR, or are refused under `--ocr=off` (see below). An image-only page inside an otherwise digital PDF is not OCR'd, but is named in `ingest_warning` |
| `.docx` | `section` | Heading styles, resolved through the document's style sheet, so localized and custom heading styles count | Headers, footers, comments and tracked-change machinery are never read. Deletions cannot leak: only `w:t` is read, never `w:delText`, and moved text is read once, where it landed |
| `.epub` | — | — | Removed, not deferred. Refused by extension, with the remedy: convert to PDF or Markdown |
| `.pptx` / `.ppt` | — | — | Removed. Run `scripts/convert-for-ingest.ps1` (Windows, PowerPoint) for a PDF plus a speaker-notes file, and ingest both. Elsewhere, `soffice --headless --convert-to pdf` gives the slides at lower fidelity and without the notes |
| `.html` | — | — | Recognised by content sniffing and refused with a reason |
| `.doc` | — | — | Legacy binary Word: refused, pointing at `scripts/convert-for-ingest.ps1`, which converts it through Word itself |

Format is decided by content, not by file extension.

### Scanned PDFs

A PDF whose sampled pages are essentially imagery — at most a quarter of them carrying text — or whose
text layer decodes to noise is routed through in-process OCR (tesseract.js, WASM, nothing to
install). Within an OCR'd document the decision is re-made per page, so a scanned book's digitally
typeset title page keeps its real text and only the scanned pages pay for recognition.

A mostly digital PDF with a few scanned pages (plates, inserts) stays on the fast path, and those
pages are not recognised. They are not silent either: the document's `ingest_warning` names every
page that yielded no text, as it does for an OCR'd page whose every line scored too low to keep.

It is slow and visibly so: roughly 1–5 seconds per page per worker, making a 400-page scan tens of
minutes. `get_document_outline` reports `chunk_count` rising against `locator_count` throughout.

| Flag | Environment variable | Default |
|---|---|---|
| `--ocr=auto\|off` | `DOCUMENT_INDEX_OCR` | `auto` |
| `--ocr-lang=` | `DOCUMENT_INDEX_OCR_LANG` | `eng` |
| `--ocr-workers=` | `DOCUMENT_INDEX_OCR_WORKERS` | `2` |
| `--ocr-lang-path=` | `DOCUMENT_INDEX_OCR_LANG_PATH` | the CDN |

The embedding model is English-only, so a multilingual library retrieves poorly. See
[docs/roadmap.md](docs/roadmap.md).

---

## The five tools

| Tool | What it does |
|---|---|
| `search_document` | Hybrid BM25 + semantic search. Ranked snippets with locators, each with a similarity score, plus a `confidence` that says when the library probably does not cover the question, and a `coverage` block naming what was not searched or has drifted. `filter.path_prefix` confines it to one folder. The usual starting point. |
| `get_document_outline` | Heading tree with chunk ranges. Also lists the library (or one folder of it, with `path_prefix`) with each file's `source_status`, and reports ingest progress. |
| `get_chunk_context` | The only tool that returns body text, hard-capped at 24,000 characters. A single chunk over the cap (possible only in an index built before October 2026's fixes) is cut and marked `truncated`. |
| `ingest_document` | Index a file. Returns immediately; indexing continues in the background. |
| `delete_document` | Drop a document from the index. Never touches the file on disk. |

The normal workflow is three cheap steps: **`search_document`** for ranked snippets, each already
naming its document and locator; **`get_document_outline`** to orient inside the one that looks right,
if needed; **`get_chunk_context`** to read the passage and its neighbours, addressed by `chunk_id`
from the hit or `document_id` + `seq` from the outline.

A `chunk_id` or `document_id` that stops resolving — the file was edited, the document deleted, the
index rebuilt — explains itself instead of reading as unknown. The index keeps a metadata-only
tombstone for every retired id: what it was, where it sat (page or section, seq) and which document
replaced it. A retired `document_id` passed to `search_document` or `get_document_outline` is followed
to the current version, and the reply says so; a retired `chunk_id` is never followed, because the
passage at the same seq in a new version may be a different one. No old text is kept, so the index
stays rebuildable from your files.

Search never returns full document bodies. That is structural rather than conventional: the output
schema for a search hit has no text field at all, so a refactor cannot quietly regress it.

Everything the reading tools return beyond their own framing comes from your files: titles, section
names, snippets, passages. Replies say so, each passage `get_chunk_context` returns is fenced so its
end is unambiguous, and the tool descriptions tell the model to treat that text as content, not
instructions. Like every MCP tool result, a `get_chunk_context` reply carries its text twice, once
rendered in `content` and once in `structuredContent`. A client that forwards both to the model pays
for the passage twice.

Reference lists are recognised at ingest (a References heading, or citation-dense text) and left out
of search by default, since a list of paper titles matches every question on its topic while
answering none. `filter.kind: "references"` searches them.

### With YouTube Transcript Notes

[YouTube Transcript Notes](https://github.com/ekelly95/youtube-transcript-notes) captures a video as
faithful, timestamped Markdown. Ingest that here and the whole chain stays checkable:

```text
YouTube video
  → YouTube Transcript Notes  → timestamped Markdown
  → Document Index MCP        → bounded passages with source locations
  → your agent                → notes or synthesis you can verify
```

The transcript's wording and clickable timestamps survive ingestion unchanged, so a claim in the final
synthesis traces back to the second of video it came from.

---

## Privacy

Documents are read from your disk and indexed into one SQLite file under your library root. Search,
ranking and passage retrieval are entirely local: no document text is sent anywhere except to the AI
client you deliberately connected.

The server makes exactly two outbound requests, both one-time downloads of its own machinery, neither
carrying any part of your documents:

- the embedding model (~65 MB) from `huggingface.co/Qdrant/bge-small-en-v1.5-onnx-Q`, on first
  ingest;
- OCR language data (~3 MB per language) from `cdn.jsdelivr.net`, on the first *scanned* PDF only. A
  library with no scans never makes this one, and `--ocr-lang-path=<dir>` removes it entirely by
  pointing at your own copy from
  [tessdata_fast](https://github.com/tesseract-ocr/tessdata_fast).

Both are cached and neither repeats. The model is checked against a pinned SHA-256 every time it is
loaded; the OCR language data is not checked. [SECURITY.md](SECURITY.md) has the details.

The library root is a jail: paths outside it are refused, symlinks that escape it are refused, and an
extension allowlist keeps files like `.env` from being addressable at all. Results expose a
library-relative path, never an absolute one — including in error messages.

One caveat worth stating plainly. The allowlist stops secrets being addressable; it does not stop a
private notes tree *inside* the root being addressable. Choose a root wide enough to hold your
documents and no wider.

---

## Development

```bash
pnpm install
pnpm build
pnpm test      # tsc, then the full suite
pnpm inspect   # MCP Inspector
```

The suite covers the chunker's boundary law, the path jail, index agreement across chunks/FTS/vectors,
PDF probe refusals and real OCR over generated scan imagery, concurrent-ingest safety and lease
recovery, hybrid fusion, and the five tools end to end over a real MCP client.

Three things to know. **Never pipe the test run** — a `| tail` once masked a failure here. **Four
tests skip by default**: two need `DOCUMENT_INDEX_TEST_REAL_MODEL=1` (with
`DOCUMENT_INDEX_MODEL_CACHE` set) — the real tokenizer's window and the confidence calibration — and
CI runs them on one job; two symlink cases skip where a symlink cannot be created, which on Windows
means without Developer Mode or an elevated shell, and CI runs them on Linux. And there are **no binary test fixtures** — PDFs are hand-assembled at
test time, cross-reference table and all, and scanned pages are drawn onto a canvas and embedded as a
JPEG XObject, so OCR is tested against genuine imagery without a blob in the repository.

[CONTRIBUTING.md](CONTRIBUTING.md) has the rest, including what will fail review.

### Retrieval evaluation

```bash
pnpm eval --library=/path/to/corpus --questions=eval/questions.json
```

Reports recall@1/3/5 and mean reciprocal rank for lexical, semantic and hybrid search separately. The
test suite proves the machinery is correct and deterministic, which is a different claim from the
ranking being good; this measures the second one, and stops a tuning change quietly regressing it.

### The stress corpus

Parser behaviour is measured against ten open-access documents, each chosen because it breaks a
different assumption. They are too large for git, so
[corpus/manifest.json](corpus/manifest.json) records every source, licence and SHA-256:

```bash
node scripts/corpus.mjs list                           # sources and re-download URLs
node scripts/corpus.mjs verify --dir=<corpus>/docs     # confirm nothing has drifted
```

A second set of six openly licensed slide decks tests the conversion route rather than a parser
(`--dir=<corpus>/decks --set=decks`). Those found two PDF-reader defects, both fixed, and confirmed
the thing worth knowing about decks: a 51-slide survey report whose chart categories and percentages
pass straight through conversion into searchable text.

---

## Design notes and limitations

The engineering record lives in `docs/`, and is worth reading before changing anything:

- **[docs/design.md](docs/design.md)** — the three ideas the architecture rests on, and every
  deliberate deviation from the original specification with its reasoning.
- **[docs/gotchas.md](docs/gotchas.md)** — the accumulated sharp edges, most of them bugs first. Each
  is a trap a reasonable change would walk straight back into.
- **[docs/roadmap.md](docs/roadmap.md)** — what is built, what was cut and why, the October 2026
  audit against a real library with its before/after numbers, and the loose ends stated honestly.

## Licence

MIT. See [LICENSE](LICENSE), and [NOTICE.md](NOTICE.md) for the dependency choices that keep it
permissive — chiefly why PDF parsing uses `pdfjs-dist` rather than the AGPL library the specification
named.

Changes are in [CHANGELOG.md](CHANGELOG.md); the security model and how to report a vulnerability are
in [SECURITY.md](SECURITY.md).