Library

Local document RAG over Postgres + pgvector, exposed to a speech assistant via MCP.

Two processes, one codebase:

• library.watcher — scans configured directories, indexes new and changed PDFs. CPU-heavy, bursty, needs outbound internet for paper metadata.

• library.server — MCP server over streamable-http. Latency-sensitive, needs only Postgres and Ollama on localhost.

They talk through Postgres LISTEN/NOTIFY, so there is no queue or extra port between them.

Setup

sudo apt install postgresql-17-pgvector
sudo apt install libvoikko1 voikko-fi   # optional: Finnish lemma/compound arm
psql -U postgres            -f sql/01_bootstrap.sql
psql -U librarian -d library -f sql/02_schema.sql
psql -U librarian -d library -f sql/03_shelf_types.sql

python3.13 -m venv .venv
.venv/bin/pip install -r requirements.txt

cp config.yaml.example config.yaml   # edit paths

Embedding server

Any OpenAI-compatible /v1/embeddings endpoint works — llama.cpp, TEI, vLLM, LM Studio — or Ollama via backend: "ollama".

With llama.cpp, run a second llama-server on its own port, separate from your chat model:

llama-server \
  -m models/bge-m3-F16.gguf \
  --embeddings \
  --pooling cls \
  -c 8192 -b 8192 -ub 8192 \
  --host 127.0.0.1 --port 8081

Or in ini-file when you can use current llama-server

[bge-m3]
model = /models/bge-m3-f16.gguf
load-on-startup = true
embeddings = on
pooling = cls
ctx-size = 8192
ubatch-size = 8192
batch-size = 8192

Then in config.yaml:

embedding:
  backend: "openai"
  base_url: "http://localhost:8081"
  model: "bge-m3"
  dimensions: 1024
  batch_size: 8

Both processes call health_check() at startup, so a wrong port, a missing --embeddings, or a dimension mismatch fails immediately rather than partway through the first index.

First index, in the foreground so you can watch it:

.venv/bin/python -m library.watcher --once -v

Then run both:

.venv/bin/python -m library.watcher      # daemon
.venv/bin/python -m library.server       # http://0.0.0.0:8765/mcp

Or install the systemd units in systemd/ (adjust User= and paths).

Related MCP server: Personal Research Assistant MCP

Adding a shelf

Append to shelves: in config.yaml and restart the watcher. The shelves table is reconciled on startup. Several shelves may share a type:

  - name: condo_meetings
    type: meetings
    path: /srv/documents/meetings/condo

Shelf types and what they do at ingest:

Pictures (JPEG)

.jpg/.jpeg files are indexed through a local vision LLM: the model describes the picture, and that description is stored as the document's text — chunked, embedded and keyword-indexed like any other document, so the ordinary search tools find pictures by what is in them. EXIF adds metadata around it: capture date becomes doc_date (so date_from/ date_to filters and date-sorting work for photos), camera model, pixel size and GPS position go into extra.

The picture itself is sent to the LLM the same way Jarvis does: base64 data:image/jpeg;base64,... inside an OpenAI-style image_url content block. Images are downscaled to max_image_px on the longest side first (a 12 MP phone photo is ~4× the vision tokens at full size).

vision:
  enabled: true
  base_url: "http://localhost:8081"   # any llama.cpp server started
  model: "qwen2.5vl"                  # with --mmproj for a vision model
  # api_key: "..."                    # only if the server requires one
  prompt: "Tell me everything about this picture."
  timeout: 120                        # seconds per picture
  max_image_px: 1568                  # long-side cap before sending
  jpeg_quality: 85

Behaviour notes:

• While enabled: false, pictures are skipped; already-captioned pictures stay in the database and remain searchable.

• A picture is re-captioned only when its bytes change (same content hash rule as other formats). The captioning cost is paid once per photo.

• A caption shorter than the chunker's min_tokens is still stored as one chunk — a picture always keeps at least its description.

• Put photos in their own shelf (any type; the picture path ignores directory-name meaning and never does network lookups):

  - name: photos
    type: meetings
    path: /home/marko/Pictures/indexed

EXIF reading needs Pillow (apt install python3-pil or pip install Pillow); without it raw JPEG bytes still go to the model, just larger and without dates/camera/GPS. A picture whose caption comes back empty (vision server misconfigured, model without image support) fails that file with a clear error and is retried on next change — it is never stored as an empty document.

Network shares and mount points

Shelves served over CIFS/NFS fail in ways a local directory never does, and the worst failure is silent: an unmounted share is just an empty directory, and a naive scan reads that as "the user deleted everything". The watcher therefore checks every shelf before it scans, and can only delete documents when the listing is trustworthy:

1. Path check — the shelf directory must exist. Gone (unmounted, exported away) → the shelf is skipped, the database is untouched.

2. Read probe — the directory is listed once under a timeout (watcher.preflight_timeout, default 30 s). A stale handle or dead server answers with ESTALE/EIO/ENOTCONN → skip; a hard mount that hangs forever times out → skip (and the scan cycle continues with the other shelves instead of freezing).

3. mount: true (per shelf) — the path must actually be under a mount point (checked against /proc/self/mountinfo). Use this for every network shelf: it catches the unmounted-but-empty-directory case that a plain read probe cannot tell apart from a real empty shelf.

4. Empty-view guard (always on) — even if all checks pass, a scan that sees zero indexable files while the database still knows documents never deletes anything and logs an error. A live mount whose export broke (renamed share, wrong subdirectory) looks exactly like this. If a shelf may genuinely go completely empty, opt in with prune_on_empty: true on that shelf.

Partial failures are safe too: a walk that dies mid-tree (mount expires while scanning) abandons the whole scan — nothing indexed, nothing deleted — and a file that merely fails to read or extract is never counted as deleted, only one that is absent from a complete listing.

shelves:
  - name: scans
    type: manuals
    path: /mnt/nas/scans
    mount: true
watcher:
  preflight_timeout: 30

Skipped shelves are logged (scan skipped, database untouched) and retried on the next cycle, so a reboot that races your systemd mount units costs nothing: the first cycles skip, indexing resumes by itself once the share is back.

MCP tools

Restricting a connection to certain shelves: restrict_to

The server accepts an HTTP query parameter restrict_to that lists the shelves a connection may use:

http://host:8765/mcp?restrict_to=manuals
http://host:8765/mcp?restrict_to=manuals&restrict_to=condo_meetings

It is an HTTP parameter of the endpoint URL, not a tool parameter: it appears in no tool schema, so the model calling the tools cannot see it, set it, or widen its own access. The person or process configuring the client connection decides the grant; the server re-reads it from the wire on every request.

• Absent — the connection sees every shelf (current behaviour).

• Present — the named shelves are the allowlist for all tools: search scopes are intersected with it, other shelves' documents are hidden by list_shelves/list_documents/find_documents and refused by get_document_info/read_document/get_context, and reindex can only target allowed shelves (a bare reindex rescans exactly the allowed ones).

• Present but empty (?restrict_to=) — nothing is accessible (fail closed). Unknown names simply never match.

Repeat the parameter or comma-separate it to allow several shelves. Access attempts outside the allowlist return an explicit error naming the allowed shelves, not silently empty results.

Web file manager (WebApp.py)

python WebApp.py            # http://host:8766/  (server.web_port)

A read-only browser page over the same database: the left panel lists the libraries (with per-shelf file counts and tick boxes) and the search box, with a mode dropdown under it; the main panel lists every matching file. An empty search lists all files of the selected libraries, so it doubles as a browsing tool.

• Hybrid (default) embeds the query and fuses a dense-vector arm with the full-text arm using the same reciprocal-rank fusion as the MCP search tools: paraphrases match, not just shared words. If the embedding server is unreachable it degrades to keyword mode and the status line says so — search never fails with the embedder down.

• Keywords only is strict PostgreSQL full-text (words AND, "exact phrase", or, not) plus a filename/title substring match. Instant, and independent of the embedding server.

Each result shows a snippet with the match highlighted and the number of matched passages; a vector-only hit (no keyword overlap) gets a plain preview of its best chunk instead of highlights.

Every result links to the real file in two ways:

• the title opens the file through this server (/file/<id>), which works in every browser and on machines without the server's mounts — PDFs open in a browser tab; text/html/.svg and friends are forced to attachment so a planted web page can never execute on this origin;

• open locally is a native file:// link, built from the shelf roots in config.yaml — it needs the same mounts on the clicking computer. Non-ASCII characters are left raw in it (Taloyhtiö, not Taloyhti%C3%B6): Firefox does not percent-decode multi-byte escapes when clicking file:// links, only when typing them, and the raw form is what its local path resolver takes. Spaces stay %20, which every browser decodes correctly here.

• the raw path is always visible with a copy button.

Notes:

• Read-only: the app never writes; nothing can be deleted or reindexed from it.

• No authentication — it exposes filenames and content snippets to whoever can reach the port. Keep it on the LAN (or behind a reverse proxy with auth), same as the MCP server.

• Library selection is remembered per browser; unknown shelf names in the URL are ignored, and selecting no libraries returns nothing.

• Firefox blocks file:// navigation from http pages by design (no about:config toggle exists). To make the plain left-click work, use the official LocalFileLinks enterprise policy for this origin only: /etc/firefox/policies/policies.json containing {"policies": {"LocalFileLinks": ["http://inferno:8766"]}}, restart Firefox, verify in about:policies. It grants only that origin the right to open known local paths — local files still cannot be read or listed by web pages.

Notes

• Retrieval is hybrid: vector and full-text arms, fused with Reciprocal Rank Fusion. Both are needed — dense vectors miss E04 and SNSW-102P16EU, keyword search misses paraphrase.

• Text search config is simple, not english, so model numbers survive and Finnish is not mangled by an English stemmer.

• The default parser keeps hyphenated words whole, so the bare word palovaroitin used to miss palo-varoitin (one hword token) and even more badly the line-broken palo-⏎varoitin PDFs produce. Two layers fix it: extraction joins line-break hyphenations (both flanking letters lowercase → joined; EU-⏎direktiivi keeps its hyphen, paragraph breaks never join) and expands PDF ligatures (fi→fi), so new and changed files store clean text for both search and embeddings; the index carries a dash/underscore-free companion column chunks.tsv_loose (generated, auto-filled by the v5 migration) searched with an identically stripped query and ranked below exact hits — legacy rows match without any re-extraction, and SNSW-102P16EU ⇄ SNSW102P16EU work in both directions. Document side and query side run the same normalisation.

• Finnish inflection is handled by a trigram arm, not stemming: Postgres' built-in Snowball Finnish stemmer cuts the base form and its inflections to different stems (palovaroitin → palovaroit, palovaroittimen → palovaroittim), so it can never match base form to inflection — measured, then rejected. Instead a third keyword arm tests trigram containment: content %> query (GIN index chunks_content_trgm, v5→v6 migration) with an exact word_similarity(query, content) >= 0.7 recheck. Base form vs inflected forms measure 0.77–1.0; unrelated Finnish words stay ≤ 0.38, so the threshold is a clean cut. The arm fires only for a single query word of ≥ 5 characters (multi-word and dashed queries keep exact+loose semantics). Scores are tiered — any exact whole-word hit outranks any trigram hit, which outranks any dash-normalized hit — with ts_rank/similarity ordering inside a tier.

• A fourth keyword arm lemmatizes with Voikko (libvoikko, voikko-fi): each chunk stores its base forms plus compound members in chunks.tsv_lemma (written at ingest, backfilled by migration v7 — no re-embedding), and a single recognised query word matches through them. This crosses what trigrams cannot: an inflected base inside an inflected compound (turvista → työttömyysturvassa, only 2/5 shared trigrams) and short words the trigram gate never fires (ovi → parvekeoven). Tier: exact > lemma > trigram > dash-loose, the bands overlapping by design — a trigram hit with near-perfect containment scores like a lemma hit, which is fair for the containment a trigram actually proves. Optional — hosts without voikko keep the column NULL and the arm stays off.

• After retrieval, an optional cross-encoder reranker (bge-reranker-v2-m3 on the same llama.cpp server, /v1/rerank) re-scores the top 40 RRF candidates jointly per query and returns the best limit. Retrieval decides who competes; the reranker decides the order. Measured on a 167-document meetings corpus: chunk-level recall@10 0.90 -> 0.97, MRR 0.58 -> 0.66. If the reranker is unreachable, search degrades to the RRF order and says so — it is never a hard dependency.

• Chunks are paragraph-packed to ~400 tokens with overlap, not split by page. The embedding text prepends the document title and the chunk's section heading (embedding only — stored content, keyword index, snippets and citations stay verbatim), so bare procedure chunks retrieve anchored. page_start/page_end are recorded as provenance; page_end > page_start means the chunk crossed a break, and the citation reads "pages 6-7".

• Supported formats beyond PDF: .docx, .xlsx/.xlsm, .ods, .txt, .md/.markdown, .pptx. A "page" is the format's provenance unit: worksheet (spreadsheets, sheet name kept in section), slide (pptx, with speaker notes), form-feed page (txt/md), or one synthetic page (a docx without explicit page breaks — cite section, not a page number). documents.extra.kind records the format. Legacy .doc/.xls are skipped (detected by OLE magic bytes, logged at debug); convert with LibreOffice or export to PDF to index them.

• Citations are returned as structured fields, not a formatted string, so the speech layer can say "page 12 of the Shelly Plus 2PM manual" without reading a filename aloud.

• Scanned PDFs with no text layer are skipped with an error. Run OCR (ocrmypdf) over them first.