# Library

Local document RAG over Postgres + pgvector, exposed to a speech assistant via MCP.

Two processes, one codebase:

- **`library.watcher`** — scans configured directories, indexes new and changed PDFs. CPU-heavy, bursty, needs outbound internet for paper metadata.
- **`library.server`** — MCP server over streamable-http. Latency-sensitive, needs only Postgres and Ollama on localhost.

They talk through Postgres `LISTEN`/`NOTIFY`, so there is no queue or extra port between them.

## Setup

```bash
sudo apt install postgresql-17-pgvector
sudo apt install libvoikko1 voikko-fi   # optional: Finnish lemma/compound arm
psql -U postgres            -f sql/01_bootstrap.sql
psql -U librarian -d library -f sql/02_schema.sql
psql -U librarian -d library -f sql/03_shelf_types.sql

python3.13 -m venv .venv
.venv/bin/pip install -r requirements.txt

cp config.yaml.example config.yaml   # edit paths
```

### Embedding server

Any OpenAI-compatible `/v1/embeddings` endpoint works — llama.cpp, TEI, vLLM,
LM Studio — or Ollama via `backend: "ollama"`.

With llama.cpp, run a **second** `llama-server` on its own port, separate from
your chat model:

```bash
llama-server \
  -m models/bge-m3-F16.gguf \
  --embeddings \
  --pooling cls \
  -c 8192 -b 8192 -ub 8192 \
  --host 127.0.0.1 --port 8081
```
Or in ini-file when you can use current llama-server
```
[bge-m3]
model = /models/bge-m3-f16.gguf
load-on-startup = true
embeddings = on
pooling = cls
ctx-size = 8192
ubatch-size = 8192
batch-size = 8192
```

Then in `config.yaml`:

```yaml
embedding:
  backend: "openai"
  base_url: "http://localhost:8081"
  model: "bge-m3"
  dimensions: 1024
  batch_size: 8
```

Both processes call `health_check()` at startup, so a wrong port, a missing
`--embeddings`, or a dimension mismatch fails immediately rather than partway
through the first index.

First index, in the foreground so you can watch it:

```bash
.venv/bin/python -m library.watcher --once -v
```

Then run both:

```bash
.venv/bin/python -m library.watcher      # daemon
.venv/bin/python -m library.server       # http://0.0.0.0:8765/mcp
```

Or install the systemd units in `systemd/` (adjust `User=` and paths).

## Adding a shelf

Append to `shelves:` in `config.yaml` and restart the watcher. The `shelves`
table is reconciled on startup. Several shelves may share a type:

```yaml
  - name: condo_meetings
    type: meetings
    path: /srv/documents/meetings/condo
```

Shelf types and what they do at ingest:

| type | parent directory means | title from | network lookup |
|---|---|---|---|
| `manuals` | device name + model | directory name | no |
| `meetings` | meeting date | filename + date | no |
| `scientific` | nothing | arXiv/Crossref, else largest font on page 1 | yes |

## Pictures (JPEG)

`.jpg`/`.jpeg` files are indexed through a local vision LLM: the model
describes the picture, and that description is stored as the document's
text — chunked, embedded and keyword-indexed like any other document, so
the ordinary search tools find pictures by what is in them. EXIF adds
metadata around it: capture date becomes `doc_date` (so `date_from`/
`date_to` filters and date-sorting work for photos), camera model, pixel
size and GPS position go into `extra`.

The picture itself is sent to the LLM the same way Jarvis does: base64
`data:image/jpeg;base64,...` inside an OpenAI-style `image_url` content
block. Images are downscaled to `max_image_px` on the longest side first
(a 12 MP phone photo is ~4× the vision tokens at full size).

```yaml
vision:
  enabled: true
  base_url: "http://localhost:8081"   # any llama.cpp server started
  model: "qwen2.5vl"                  # with --mmproj for a vision model
  # api_key: "..."                    # only if the server requires one
  prompt: "Tell me everything about this picture."
  timeout: 120                        # seconds per picture
  max_image_px: 1568                  # long-side cap before sending
  jpeg_quality: 85
```

Behaviour notes:

* While `enabled: false`, pictures are skipped; already-captioned pictures
  stay in the database and remain searchable.
* A picture is re-captioned only when its bytes change (same content hash
  rule as other formats). The captioning cost is paid once per photo.
* A caption shorter than the chunker's `min_tokens` is still stored as one
  chunk — a picture always keeps at least its description.
* Put photos in their own shelf (any type; the picture path ignores
  directory-name meaning and never does network lookups):

```yaml
  - name: photos
    type: meetings
    path: /home/marko/Pictures/indexed
```

EXIF reading needs Pillow (`apt install python3-pil` or
`pip install Pillow`); without it raw JPEG bytes still go to the model,
just larger and without dates/camera/GPS. A picture whose caption comes
back empty (vision server misconfigured, model without image support)
fails that file with a clear error and is retried on next change — it is
never stored as an empty document.

## Network shares and mount points

Shelves served over CIFS/NFS fail in ways a local directory never does,
and the worst failure is silent: an unmounted share is just an empty
directory, and a naive scan reads that as "the user deleted everything".
The watcher therefore checks every shelf before it scans, and can only
delete documents when the listing is trustworthy:

1. **Path check** — the shelf directory must exist. Gone (unmounted,
   exported away) → the shelf is skipped, the database is untouched.
2. **Read probe** — the directory is listed once under a timeout
   (`watcher.preflight_timeout`, default 30 s). A stale handle or dead
   server answers with ESTALE/EIO/ENOTCONN → skip; a hard mount that
   hangs forever times out → skip (and the scan cycle continues with the
   other shelves instead of freezing).
3. **`mount: true`** (per shelf) — the path must actually be under a
   mount point (checked against `/proc/self/mountinfo`). Use this for
   every network shelf: it catches the unmounted-but-empty-directory case
   that a plain read probe cannot tell apart from a real empty shelf.
4. **Empty-view guard** (always on) — even if all checks pass, a scan
   that sees zero indexable files while the database still knows
   documents never deletes anything and logs an error. A live mount whose
   *export* broke (renamed share, wrong subdirectory) looks exactly like
   this. If a shelf may genuinely go completely empty, opt in with
   `prune_on_empty: true` on that shelf.

Partial failures are safe too: a walk that dies mid-tree (mount expires
while scanning) abandons the whole scan — nothing indexed, nothing
deleted — and a file that merely fails to *read or extract* is never
counted as deleted, only one that is absent from a complete listing.

```yaml
shelves:
  - name: scans
    type: manuals
    path: /mnt/nas/scans
    mount: true
watcher:
  preflight_timeout: 30
```

Skipped shelves are logged (`scan skipped, database untouched`) and
retried on the next cycle, so a reboot that races your systemd mount
units costs nothing: the first cycles skip, indexing resumes by itself
once the share is back.

## MCP tools

| tool | purpose |
|---|---|
| `list_shelves` | orientation: what exists, how big |
| `list_documents` | enumerate a small/medium shelf, optionally date-filtered |
| `find_documents` | semantic search over titles + abstracts, for large shelves |
| `search_manuals` | full-text + vector over manuals, `device` filter |
| `search_meetings` | same over meetings, with `date_from`/`date_to`/`dates` |
| `search_papers` | same over papers, with `author`/`year_from`/`year_to` |
| `get_context` | neighbouring passages when a result is truncated |
| `get_document_info` | full metadata for one document |
| `reindex` | signal the watcher to rescan |

## Restricting a connection to certain shelves: `restrict_to`

The server accepts an HTTP query parameter `restrict_to` that lists the
shelves a connection may use:

```
http://host:8765/mcp?restrict_to=manuals
http://host:8765/mcp?restrict_to=manuals&restrict_to=condo_meetings
```

It is an HTTP parameter of the endpoint URL, not a tool parameter: it
appears in no tool schema, so the model calling the tools cannot see it,
set it, or widen its own access. The person or process configuring the
client connection decides the grant; the server re-reads it from the wire
on every request.

- **Absent** — the connection sees every shelf (current behaviour).
- **Present** — the named shelves are the allowlist for all tools: search
  scopes are intersected with it, other shelves' documents are hidden by
  `list_shelves`/`list_documents`/`find_documents` and refused by
  `get_document_info`/`read_document`/`get_context`, and `reindex` can
  only target allowed shelves (a bare `reindex` rescans exactly the
  allowed ones).
- **Present but empty** (`?restrict_to=`) — nothing is accessible
  (fail closed). Unknown names simply never match.

Repeat the parameter or comma-separate it to allow several shelves. Access
attempts outside the allowlist return an explicit error naming the allowed
shelves, not silently empty results.

## Web file manager (WebApp.py)

```
python WebApp.py            # http://host:8766/  (server.web_port)
```

A read-only browser page over the same database: the left panel lists the
libraries (with per-shelf file counts and tick boxes) and the search
box, with a mode dropdown under it; the main panel lists every matching
file. An empty search lists all files of the selected libraries, so it
doubles as a browsing tool.

* **Hybrid** (default) embeds the query and fuses a dense-vector arm
  with the full-text arm using the same reciprocal-rank fusion as the
  MCP search tools: paraphrases match, not just shared words. If the
  embedding server is unreachable it degrades to keyword mode and the
  status line says so — search never fails with the embedder down.
* **Keywords only** is strict PostgreSQL full-text (`words` AND,
  `"exact phrase"`, `or`, `not`) plus a filename/title substring match.
  Instant, and independent of the embedding server.

Each result shows a snippet with the match highlighted and the number of
matched passages; a vector-only hit (no keyword overlap) gets a plain
preview of its best chunk instead of highlights.

Every result links to the **real file** in two ways:

* the **title** opens the file *through this server* (`/file/<id>`), which
  works in every browser and on machines without the server's mounts —
  PDFs open in a browser tab; `text/html`/`.svg` and friends are forced
  to `attachment` so a planted web page can never execute on this origin;
* **open locally** is a native `file://` link, built from the shelf roots
  in `config.yaml` — it needs the same mounts on the clicking computer.
  Non-ASCII characters are left raw in it (`Taloyhtiö`, not
  `Taloyhti%C3%B6`): Firefox does not percent-decode multi-byte escapes
  when clicking `file://` links, only when typing them, and the raw form
  is what its local path resolver takes. Spaces stay `%20`, which every
  browser decodes correctly here.
* the raw path is always visible with a copy button.

Notes:

* Read-only: the app never writes; nothing can be deleted or reindexed
  from it.
* No authentication — it exposes filenames and content snippets to
  whoever can reach the port. Keep it on the LAN (or behind a reverse
  proxy with auth), same as the MCP server.
* Library selection is remembered per browser; unknown shelf names in
  the URL are ignored, and selecting no libraries returns nothing.
* Firefox blocks `file://` navigation from http pages by design (no
  about:config toggle exists). To make the plain left-click work, use the
  official `LocalFileLinks` enterprise policy for this origin only:
  `/etc/firefox/policies/policies.json` containing
  `{"policies": {"LocalFileLinks": ["http://inferno:8766"]}}`, restart
  Firefox, verify in `about:policies`. It grants only that origin the
  right to open known local paths — local files still cannot be read or
  listed by web pages.

## Notes

- Retrieval is hybrid: vector and full-text arms, fused with Reciprocal Rank
  Fusion. Both are needed — dense vectors miss `E04` and `SNSW-102P16EU`,
  keyword search misses paraphrase.
- Text search config is `simple`, not `english`, so model numbers survive and
  Finnish is not mangled by an English stemmer.
- The default parser keeps hyphenated words whole, so the bare word
  `palovaroitin` used to miss `palo-varoitin` (one `hword` token) and even
  more badly the line-broken `palo-`⏎`varoitin` PDFs produce. Two layers fix
  it: **extraction** joins line-break hyphenations (both flanking letters
  lowercase → joined; `EU-`⏎`direktiivi` keeps its hyphen, paragraph breaks
  never join) and expands PDF ligatures (`fi`→`fi`), so new and changed files
  store clean text for both search and embeddings; the **index** carries a
  dash/underscore-free companion column `chunks.tsv_loose` (generated,
  auto-filled by the v5 migration) searched with an identically stripped
  query and ranked below exact hits — legacy rows match without any
  re-extraction, and `SNSW-102P16EU` ⇄ `SNSW102P16EU` work in both
  directions. Document side and query side run the same normalisation.
- Finnish inflection is handled by a trigram arm, not stemming: Postgres'
  built-in Snowball Finnish stemmer cuts the base form and its inflections
  to *different* stems (`palovaroitin` → `palovaroit`, `palovaroittimen` →
  `palovaroittim`), so it can never match base form to inflection —
  measured, then rejected. Instead a third keyword arm tests trigram
  containment: `content %> query` (GIN index `chunks_content_trgm`, v5→v6
  migration) with an exact `word_similarity(query, content) >= 0.7`
  recheck. Base form vs inflected forms measure 0.77–1.0; unrelated
  Finnish words stay ≤ 0.38, so the threshold is a clean cut. The arm
  fires only for a single query word of ≥ 5 characters (multi-word and
  dashed queries keep exact+loose semantics). Scores are tiered —
  any exact whole-word hit outranks any trigram hit, which outranks any
  dash-normalized hit — with `ts_rank`/similarity ordering inside a tier.
- A fourth keyword arm lemmatizes with **Voikko** (libvoikko, `voikko-fi`):
  each chunk stores its base forms plus compound members in
  `chunks.tsv_lemma` (written at ingest, backfilled by migration v7 — no
  re-embedding), and a single recognised query word matches through them.
  This crosses what trigrams cannot: an inflected base inside an inflected
  compound (`turvista` → `työttömyysturvassa`, only 2/5 shared trigrams)
  and short words the trigram gate never fires (`ovi` → `parvekeoven`).
  Tier: exact > lemma > trigram > dash-loose, the bands overlapping by
  design — a trigram hit with near-perfect containment scores like a
  lemma hit, which is fair for the containment a trigram actually
  proves. Optional — hosts without
  voikko keep the column NULL and the arm stays off.
- After retrieval, an optional cross-encoder reranker (`bge-reranker-v2-m3`
  on the same llama.cpp server, `/v1/rerank`) re-scores the top 40 RRF
  candidates jointly per query and returns the best `limit`. Retrieval
  decides *who* competes; the reranker decides *the order*. Measured on a
  167-document meetings corpus: chunk-level recall@10 0.90 -> 0.97,
  MRR 0.58 -> 0.66. If the reranker is unreachable, search degrades to
  the RRF order and says so — it is never a hard dependency.
- Chunks are paragraph-packed to ~400 tokens with overlap, not split by page.
  The embedding text prepends the document title and the chunk's section
  heading (embedding only — stored content, keyword index, snippets and
  citations stay verbatim), so bare procedure chunks retrieve anchored.
  `page_start`/`page_end` are recorded as provenance; `page_end > page_start`
  means the chunk crossed a break, and the citation reads "pages 6-7".
- Supported formats beyond PDF: `.docx`, `.xlsx`/`.xlsm`, `.ods`, `.txt`,
  `.md`/`.markdown`, `.pptx`. A "page" is the format's provenance unit:
  worksheet (spreadsheets, sheet name kept in `section`), slide (pptx, with
  speaker notes), form-feed page (txt/md), or one synthetic page (a docx
  without explicit page breaks — cite `section`, not a page number).
  `documents.extra.kind` records the format. Legacy `.doc`/`.xls` are skipped
  (detected by OLE magic bytes, logged at debug); convert with LibreOffice
  or export to PDF to index them.
- Citations are returned as structured fields, not a formatted string, so the
  speech layer can say "page 12 of the Shelly Plus 2PM manual" without reading
  a filename aloud.
- Scanned PDFs with no text layer are skipped with an error. Run OCR
  (`ocrmypdf`) over them first.