file-indexer

An HTTP MCP server that breaks large documents into an exact-line-numbered section index, so a small local model can locate, read, summarize and edit one part of a document at a time — without ever ingesting the whole file.

Formats: markdown, plain text, HTML, PDF.

Why

A 207k-token document does not fit in a 32k-context model. Uploading it is not the answer; finding the right 1,600 tokens is. The index is built by plain extraction — never by the model — so it is exact and effectively free:

745,247 chars / 207k tokens / 8,079 lines  →  129 sections in 234ms, largest 1,666 tokens
find("tslib")  →  section s112, lines 6971-7034, ~1,610 tokens   (zero model tokens spent)

Retrieval is BM25 over the index and costs nothing. The model only ever sees the span it asked for.

How a model uses it

Question	Tool	Cost
What's in this file?	`outline`	0 tokens
Where is X?	`find`	0 tokens
Read just that part	`read_section` / `read_span`	one section
Process the whole file part by part	`next_unit`	one unit per call
Summarize the file	`summarize_file`	map-reduce, reports `coverage`
Add something at section X	`edit_section`	backed up, restorable

Sections carry both own and subtree sizes, so a model can tell a 32-token heading from the 11k-token chapter beneath it before committing to a read. Documents with no headings still get a navigable index: HTML falls back to <section>/<article>, then batched repeating rows, then synthetic packing — such sections are marked synthetic and shown as (auto).

summarize_file returns a coverage fraction and an explicit capped/dropped_sections list. A summary that silently skips most of a document reads exactly like a good one, so under-coverage is reported rather than hidden.

Tools

Tool	Description
`index_file`	Build or refresh one document's index; incremental, milliseconds
`index_all`	Index every supported document under the workspace
`index_status`	Whether a document is indexed, and whether it changed on disk since
`list_documents`	Indexable documents with size and indexed state
`outline`	Table of contents: section ids, depth, titles, line spans, sizes
`find`	Locate sections by keyword — zero-token BM25, no model call
`read_section`	Read one section's own text (or one chunk of it)
`read_span`	Read an arbitrary line range verbatim
`next_unit`	Walk a document sequentially, one readable unit per call
`summarize_section`	Summarize one section or its subtree; cached
`summarize_file`	Whole-document map-reduce summary with a coverage figure
`edit_section`	Insert/replace/delete at a section id — 7 explicit modes
`edit_replace`	Unique-match search/replace, with a whitespace-tolerant fallback
`list_backups` / `restore_backup`	Every write is backed up first
`health`	Model endpoint, configured model, indexed document count

edit_section modes are explicit about subsections, because picking the wrong boundary silently misplaces text: append (end of own text, before subsections), append_subtree (after all of them), prepend, replace_body, replace_subtree, replace_all, delete.

PDF

A PDF has no lines. Its text is extracted once to a sidecar under .fileindex/text/, and every line number reported for a PDF refers to that extracted text, not the PDF. Sections also carry start_page/end_page:

s2  #  Preface           (lines 47-121, ~1122 tok, p.4-5)
s7  #  List of Figures   (lines 490-662, ~2493 tok, p.15-18) [2 chunks]

Structure comes from PDF bookmarks when present, then a font-size heuristic, then page grouping. PDF is read-only — the edit tools refuse it.

Run it

cp .env.example .env      # set WORKSPACE_HOST_DIR to the directory you want indexed
./start.sh

start.sh runs the container as the owner of WORKSPACE_HOST_DIR. This matters: the editor writes atomically (temp file + rename), so a root container would rewrite an edited document as root:root and lock its owner out of their own file.

The workspace is mounted read-write — edit_section/edit_replace can create, modify or delete anything under it. Scope it to the documents you actually want editable.

Summarization uses a local model over an OpenAI-compatible endpoint (llama.cpp/llama-swap by default, Ollama optional). Point LLAMACPP_URL at an already-loaded instance rather than letting a second process load its own copy — two copies of the same model will exhaust a small GPU.

Adding a format

parsers.py is a registry mapping extension to a parser returning list[dict] sections shaped {title, depth, start_line, end_line, heading_lines}. A new line-based format is one module plus one registry entry, and inherits the heading-less fallback for free.

PDF is the exception: it has no lines, so parsers.extract_pdf() does extraction and section detection together and returns text that docindex.build() writes to a sidecar and indexes instead.

Licence

MIT — see LICENSE.

PDF extraction uses PyMuPDF, which is AGPL-3.0. That only matters if you redistribute this server; it is not a concern for local or private-network use. Everything else is the Python standard library plus the MCP SDK.

file-indexer-mcp