The Librarian

Agentic RAG over a private podcast archive, measured before it was trusted.

Portfolio exhibit. This is a sanitized public extract of a private system in daily use. The architecture and method are real; the data and identifiers are stand-ins, and the section "What ships here, and what is sanitized" below lists which is which.

What it does

The Librarian answers questions over a 55-episode podcast archive: research notes plus roughly 19 hours of audio, transcribed with timestamps by Whisper and chunked into about 2,200 addressable passages. Each question runs through two retrieval branches, lexical and semantic, fused into one ranking that cites the exact passage and minute. A 122-query gold set scores recall@k and MRR across four strategies, offline, with one command.

▶ Watch the demos

Related MCP server: confluence

The eval is the signal

Retrieval quality is a measurable problem, so it gets measured. eval/gold_set.json holds 122 paraphrase queries, two per episode, each written against one target chunk and worded to avoid that chunk's surface tokens, so the eval scores paraphrase retrieval rather than string matching. eval/run.py scores recall@1, recall@3, recall@5, and MRR for four retrieval strategies over the 545-chunk sample corpus and prints the table live. Nothing is hard-coded.

It runs offline with nothing but the Python standard library:

python eval/run.py

Output from the default offline run in this repo:

corpus: 545 chunks   gold queries: 122
dense backend: fallback-char-ngram
(offline fallback dense vectors; set USE_ST=1 for real embeddings)

Retrieval mode         | R@1    | R@3    | R@5    | MRR
-----------------------+--------+--------+--------+-------
BM25, raw tokens       | 0.47   | 0.60   | 0.64   | 0.54
BM25, normalized       | 0.50   | 0.66   | 0.71   | 0.58
Dense, semantic        | 0.25   | 0.34   | 0.42   | 0.31
Hybrid, RRF            | 0.42   | 0.57   | 0.66   | 0.51

best MRR: BM25, normalized (0.58)

Those numbers are gated in CI: every push recomputes the table and fails the build if any tracked metric falls below its floor, if fusion stops improving on its weaker branch, or if the table above drifts from a live run. The floors sit four to five points under the measured values, about six flipped queries out of 122, so ordinary noise passes and a real regression fails.

An earlier version of this eval ran 12 queries over 15 chunks. On a corpus that small every strategy lands near 1.00 and the table mostly proves the wiring, so the corpus was scaled until the numbers could disagree. At 545 chunks they do. Normalized BM25 recovers inflected matches that raw tokens miss (recall@3 0.66 against 0.60). The offline fallback vectors collapse at this scale (MRR 0.31), as surface n-grams should on paraphrase queries. And fusing that weak branch in drags the hybrid below plain normalized BM25 (MRR 0.51 against 0.58). A third of the queries put nothing relevant in the lexical top three, which is what the paraphrase wording is there to force.

With USE_ST=1 the dense branch swaps the fallback for real multilingual-e5 embeddings. Same corpus, same gold set, measured by hand (the model is a download, so CI stays on the offline path):

Retrieval mode         | R@1    | R@3    | R@5    | MRR
-----------------------+--------+--------+--------+-------
BM25, raw tokens       | 0.47   | 0.60   | 0.64   | 0.54
BM25, normalized       | 0.50   | 0.66   | 0.71   | 0.58
Dense, semantic        | 0.73   | 0.89   | 0.94   | 0.81
Hybrid, RRF            | 0.61   | 0.81   | 0.92   | 0.72

The same lesson from the other side: real embeddings win the table outright (MRR 0.81), and RRF fusion with the much weaker lexical branch gives part of that lead back (0.72). At this corpus size, whichever branch is stronger, fusing in a much weaker partner costs rank quality. The 15-chunk eval showed hybrid on top and hid all of this; scaling the corpus is what gave the eval the resolution to catch it, and catching it is what an eval is for.

What ships here, and what is sanitized

Real: the architecture and the measurement. The two-branch retrieval, the RRF fusion, the gold-set eval, the recall@k and MRR scoring, and the typed MCP surface run end to end in this repo.

Everything below is a deliberate, labeled stand-in, so the pipeline runs offline with zero downloads while the real corpus stays private:

• Corpus. The shipped sample is 545 timestamped chunks across 55 episodes, roughly a quarter of the full corpus size. The passages are synthetic English stand-ins written for this repo: they mirror the archive's shape (episode structure, notes and transcript layers, timestamps, podcast-register topics); none of the real Czech episode content appears, and neither do guest or host names.

• Normalization. The full system uses a real Czech lemmatizer (simplemma); the sample ships a small standard-library suffix stemmer that reproduces the effect that matters for the score, matching inflected forms to a shared stem.

• Embeddings. The full dense branch runs multilingual-e5 locally; the shipped eval defaults to an offline char-n-gram vector fallback, labeled as such at every boundary. USE_ST=1 turns the real model on.

• Not in this repo. Whisper transcription and the grounded answer-generation layer. This repo measures the retrieval beneath them.

Demos

Three short clips, recorded live against the sample corpus, one capability each.

Audio-only knowledge. A fact that exists only in the spoken transcript: how fast electrons actually drift versus how fast the signal travels. Retrieved and cited to the minute.

https://github.com/user-attachments/assets/4ab15dd1-6891-4f4c-b023-9ac0a22b21df

A concept spoken, never written down. The name for the mythic descent into the underworld appears only in the audio; a paraphrased question sharing none of its surface words still lands on the right passage.

https://github.com/user-attachments/assets/02ac8608-871e-482e-bc17-3e5ca339227f

Cross-episode synthesis. An answer spread across more than one episode; retrieval pulls the passages from each, and every claim keeps its own timestamped citation.

https://github.com/user-attachments/assets/a755d3ce-929d-4ae7-bfe5-3a7ca8569e7a

How it works

• Two branches, one fused ranking. BM25 scores lexical overlap, the dense branch semantic similarity; reciprocal rank fusion (constant c=60, from Cormack et al.) combines their ranks, since the raw scores share no scale.

• One place defines every branch. retrieval/pipeline.py builds bm25_raw, bm25_stem, dense, and hybrid once; the eval and the MCP server can never drift apart on what those modes mean.

• Grounded, addressable answers. Every chunk carries its episode, source layer (notes vs transcript), and start time; each result returns a citation tag like Ep. ep03 @ 08:03.

flowchart TB
    NOTES["Research notes, 55 episodes"] --> CHUNK
    AUDIO["Audio, 19 h, Whisper timestamped transcripts"] --> CHUNK["Timestamped chunks<br/>~2200 full, 545 in this sample"]
    CHUNK --> BM25["BM25 lexical<br/>raw + normalized"]
    CHUNK --> DENSE["Dense embeddings<br/>multilingual-e5 or offline fallback"]
    BM25 --> RRF["Reciprocal rank fusion, c=60"]
    DENSE --> RRF
    RRF --> MCP["MCP server, 3 typed tools<br/>ranked, cited passages"]
    RRF --> EVAL["Eval, recall@k and MRR<br/>gold set, 122 queries"]
    MCP --> ANS["Grounded answer, cited to the minute<br/>answer layer not in this repo"]

The MCP surface

A FastMCP server exposes the exact pipeline the eval measures through three typed tools; the numbers above describe what an agent receives.

python mcp_server.py            # stdio transport
MCP_HTTP=1 python mcp_server.py # streamable HTTP on 127.0.0.1:8765

• search_podcast(query, k=5, mode="hybrid") returns ranked, timestamped, cited passages. mode is one of bm25_raw, bm25_stem, dense, hybrid.

• list_episodes() returns the episode catalog with per-episode chunk counts.

• episode_detail(episode_id) returns one episode's chunks in timestamp order.

Stack

The default python eval/run.py install is empty: standard library only. mcp>=1.2.0 runs the server; sentence-transformers, simplemma, and faster-whisper are optional, for the real models.

Status and contact

PRODUCTION EXTRACT. A sanitized public cut of a private system in real use. I direct AI coding tools to build it; the architecture, the fusion design, the gold set, and the eval methodology are mine.

More of the portfolio at github.com/janvrsinsky.

• LinkedIn: linkedin.com/in/janvrsinsky

Topics