# file-indexer

An HTTP MCP server that breaks large documents into an exact-line-numbered section index, so a
small local model can locate, read, summarize and edit one part of a document at a time — without
ever ingesting the whole file.

Formats: **markdown, plain text, HTML, PDF**.

## Why

A 207k-token document does not fit in a 32k-context model. Uploading it is not the answer; finding
the right 1,600 tokens is. The index is built by plain extraction — never by the model — so it is
exact and effectively free:

```
745,247 chars / 207k tokens / 8,079 lines  →  129 sections in 234ms, largest 1,666 tokens
find("tslib")  →  section s112, lines 6971-7034, ~1,610 tokens   (zero model tokens spent)
```

Retrieval is BM25 over the index and costs nothing. The model only ever sees the span it asked for.

## How a model uses it

| Question | Tool | Cost |
|---|---|---|
| What's in this file? | `outline` | 0 tokens |
| Where is X? | `find` | 0 tokens |
| Read just that part | `read_section` / `read_span` | one section |
| Process the whole file part by part | `next_unit` | one unit per call |
| Summarize the file | `summarize_file` | map-reduce, reports `coverage` |
| Add something at section X | `edit_section` | backed up, restorable |

Sections carry both **own** and **subtree** sizes, so a model can tell a 32-token heading from the
11k-token chapter beneath it before committing to a read. Documents with no headings still get a
navigable index: HTML falls back to `<section>`/`<article>`, then batched repeating rows, then
synthetic packing — such sections are marked `synthetic` and shown as `(auto)`.

`summarize_file` returns a `coverage` fraction and an explicit `capped`/`dropped_sections` list.
A summary that silently skips most of a document reads exactly like a good one, so under-coverage
is reported rather than hidden.

## Tools

| Tool | Description |
|---|---|
| `index_file` | Build or refresh one document's index; incremental, milliseconds |
| `index_all` | Index every supported document under the workspace |
| `index_status` | Whether a document is indexed, and whether it changed on disk since |
| `list_documents` | Indexable documents with size and indexed state |
| `outline` | Table of contents: section ids, depth, titles, line spans, sizes |
| `find` | Locate sections by keyword — zero-token BM25, no model call |
| `read_section` | Read one section's own text (or one chunk of it) |
| `read_span` | Read an arbitrary line range verbatim |
| `next_unit` | Walk a document sequentially, one readable unit per call |
| `summarize_section` | Summarize one section or its subtree; cached |
| `summarize_file` | Whole-document map-reduce summary with a coverage figure |
| `edit_section` | Insert/replace/delete at a section id — 7 explicit modes |
| `edit_replace` | Unique-match search/replace, with a whitespace-tolerant fallback |
| `list_backups` / `restore_backup` | Every write is backed up first |
| `health` | Model endpoint, configured model, indexed document count |

`edit_section` modes are explicit about subsections, because picking the wrong boundary silently
misplaces text: `append` (end of own text, before subsections), `append_subtree` (after all of
them), `prepend`, `replace_body`, `replace_subtree`, `replace_all`, `delete`.

## PDF

A PDF has no lines. Its text is extracted once to a sidecar under `.fileindex/text/`, and **every
line number reported for a PDF refers to that extracted text**, not the PDF. Sections also carry
`start_page`/`end_page`:

```
s2  #  Preface           (lines 47-121, ~1122 tok, p.4-5)
s7  #  List of Figures   (lines 490-662, ~2493 tok, p.15-18) [2 chunks]
```

Structure comes from PDF bookmarks when present, then a font-size heuristic, then page grouping.
PDF is **read-only** — the edit tools refuse it.

## Run it

```
cp .env.example .env      # set WORKSPACE_HOST_DIR to the directory you want indexed
./start.sh
```

`start.sh` runs the container as the owner of `WORKSPACE_HOST_DIR`. This matters: the editor writes
atomically (temp file + rename), so a root container would rewrite an edited document as `root:root`
and lock its owner out of their own file.

The workspace is mounted **read-write** — `edit_section`/`edit_replace` can create, modify or delete
anything under it. Scope it to the documents you actually want editable.

Summarization uses a local model over an OpenAI-compatible endpoint (llama.cpp/llama-swap by
default, Ollama optional). Point `LLAMACPP_URL` at an *already-loaded* instance rather than letting
a second process load its own copy — two copies of the same model will exhaust a small GPU.

## Adding a format

`parsers.py` is a registry mapping extension to a parser returning
`list[dict]` sections shaped `{title, depth, start_line, end_line, heading_lines}`. A new line-based
format is one module plus one registry entry, and inherits the heading-less fallback for free.

PDF is the exception: it has no lines, so `parsers.extract_pdf()` does extraction and section
detection together and returns text that `docindex.build()` writes to a sidecar and indexes instead.

## Licence

MIT — see [LICENSE](LICENSE).

PDF extraction uses [PyMuPDF](https://pypi.org/project/PyMuPDF/), which is **AGPL-3.0**. That only
matters if you redistribute this server; it is not a concern for local or private-network use.
Everything else is the Python standard library plus the MCP SDK.