The Librarian

Agentic RAG over a private podcast archive, measured before it was trusted. Two retrieval branches, lexical and semantic, fuse into a single ranking that cites every answer down to the minute. The load-bearing part is not the retrieval, it is the eval underneath it: a hand-labeled gold set scoring recall@k and MRR across keyword, dense, and hybrid, runnable offline with one command.

What it is

The Librarian answers questions over a podcast archive of 55 episodes: written research notes for each one, plus roughly 19 hours of audio turned into timestamped transcripts by Whisper. Notes and transcripts are chunked into about 2,200 addressable passages, each tagged with its episode and start time. A question fans out into two retrieval branches at once, a lexical one and a semantic one, whose rankings are fused so the answer can point back at the exact passage and minute it came from. Knowledge that was only ever spoken aloud becomes retrievable and citable.

The claim I actually care about is not "it retrieves." It is "here is how well, and here is how I measured it." That is the part most RAG demos skip, so it leads here.

Related MCP server: RAG In A Box MCP Server

The eval is the signal

Retrieval quality is a measurable problem, so it gets measured. eval/gold_set.json is a hand-labeled set of 12 paraphrase queries, each worded to avoid the target passage's surface tokens so the eval scores paraphrase retrieval rather than string matching. eval/run.py scores recall@1, recall@3, recall@5, and MRR for four retrieval strategies and prints the table live from the corpus and the gold set. Nothing is hard-coded.

It runs offline with nothing but the Python standard library:

python eval/run.py

Output from the default offline run in this repo:

corpus: 15 chunks   gold queries: 12
dense backend: fallback-char-ngram
(offline fallback dense vectors; set USE_ST=1 for real embeddings)

Retrieval mode         | R@1    | R@3    | R@5    | MRR
-----------------------+--------+--------+--------+-------
BM25, raw tokens       | 0.67   | 0.92   | 0.92   | 0.79
BM25, normalized       | 0.58   | 1.00   | 1.00   | 0.79
Dense, semantic        | 0.75   | 1.00   | 1.00   | 0.86
Hybrid, RRF            | 0.75   | 1.00   | 1.00   | 0.88

best MRR: Hybrid, RRF (0.88)

The crossover is the point. Raw keyword search leaves relevant passages out of reach on an inflected corpus; normalizing tokens trades a little rank-1 precision for full recall by rank 3. The dense branch leads on early precision, and reciprocal rank fusion keeps that precision while edging MRR to the top of the four. These are deliberately small-set numbers: read the separation between strategies, not the decimals. The harness is built to make that separation visible and reproducible, and to keep reporting it honestly as the corpus and the models change.

Set USE_ST=1 to swap the offline fallback for real multilingual embeddings and re-run the same table.

What ships here, and what is sanitized

Being precise about this, because a retrieval claim is only worth as much as its honesty about the setup.

Real: the architecture and the measurement. The two-branch retrieval, the RRF fusion, the gold-set eval, the recall@k and MRR scoring, and the typed MCP surface are the actual engineering, and they run end to end in this repo.

Everything below is a deliberate, labeled stand-in so the whole pipeline runs offline with zero downloads, while the real corpus stays private:

• Corpus. Only a representative sample of the archive ships: 15 timestamped chunks across 5 episodes. The chunk texts are English stand-ins for the Czech originals, written so the retrieval trace is readable to anyone, and guest and host names are redacted. The real episode corpora are private and stay private.

• Normalization. The full system normalizes with a real Czech lemmatizer (simplemma); the shipped sample uses a small standard-library suffix stemmer as a stand-in, so the eval runs offline. It reproduces the effect that matters for the score (matching inflected forms to a shared stem) without the dependency.

• Embeddings. The full dense branch is multilingual-e5 run locally; the shipped eval uses an offline char-n-gram vector fallback by default. It is clearly not a neural embedding, and it is labeled as such at every boundary. The real model turns on with USE_ST=1.

• Not in this repo. Whisper transcription and the grounded answer-generation layer are not included. This repo measures the retrieval that sits beneath them, which is where a RAG system is made or broken.

Demos

Three short clips, recorded live against the sample corpus, one capability each.

Audio-only knowledge. A fact that lives only in the spoken transcript, not the notes: how fast electrons actually drift versus how fast the signal travels. The retriever surfaces it and cites the minute.

https://github.com/user-attachments/assets/4ab15dd1-6891-4f4c-b023-9ac0a22b21df

A concept spoken, never written down. The name for the mythic descent into the underworld appears only in the audio. A paraphrased question that shares none of its surface words still lands on the right passage.

https://github.com/user-attachments/assets/02ac8608-871e-482e-bc17-3e5ca339227f

Cross-episode synthesis. A question whose answer is spread across more than one episode. Retrieval pulls the passages from each and every claim keeps its own timestamped citation.

https://github.com/user-attachments/assets/a755d3ce-929d-4ae7-bfe5-3a7ca8569e7a

How it works

The system is a short list of design decisions, not a framework.

• Two branches, one fused ranking. BM25 scores lexical overlap; the dense branch scores semantic similarity. Their scores are not on a comparable scale, but their ranks are, so reciprocal rank fusion (constant c=60, from Cormack et al.) combines them without either branch dominating.

• One place defines every branch. retrieval/pipeline.py builds bm25_raw, bm25_stem, dense, and hybrid once, so the eval and the MCP server can never drift apart on what those modes mean. What the eval measures is exactly what an agent gets served.

• Grounded, addressable answers. Every chunk carries its episode, source layer (notes vs transcript), and start time, so each result comes back with a citation tag like Ep. ep03 @ 08:03. The answer can always point at where it came from.

• Offline by default, real models on a flag. The eval depends on nothing but the standard library. Optional accelerators (rank-bm25) and real models (multilingual-e5) are opt-in, never required to read the design or reproduce the table.

flowchart TB
    NOTES["Research notes, 55 episodes"] --> CHUNK
    AUDIO["Audio, 19 h, Whisper timestamped transcripts"] --> CHUNK["Timestamped chunks<br/>~2200 full, 15 in this sample"]
    CHUNK --> BM25["BM25 lexical<br/>raw + normalized"]
    CHUNK --> DENSE["Dense embeddings<br/>multilingual-e5 or offline fallback"]
    BM25 --> RRF["Reciprocal rank fusion, c=60"]
    DENSE --> RRF
    RRF --> MCP["MCP server, 3 typed tools<br/>ranked, cited passages"]
    RRF --> EVAL["Eval, recall@k and MRR<br/>gold set, 12 queries"]
    MCP --> ANS["Grounded answer, cited to the minute<br/>answer layer not in this repo"]

The MCP surface

The retrieval is exposed to an agent through a FastMCP server with three small, typed tools. It serves the exact pipeline the eval measures, so the numbers above describe what an agent actually receives.

python mcp_server.py            # stdio transport
MCP_HTTP=1 python mcp_server.py # streamable HTTP on 127.0.0.1:8765

• search_podcast(query, k=5, mode="hybrid") returns ranked, timestamped, cited passages. mode is one of bm25_raw, bm25_stem, dense, hybrid.

• list_episodes() returns the episode catalog with per-episode chunk counts.

• episode_detail(episode_id) returns one episode's chunks in timestamp order.

Stack

The default python eval/run.py install is empty: standard library only. mcp>=1.2.0 is needed only to run the server; rank-bm25, sentence-transformers, simplemma, and faster-whisper are optional, for acceleration or the real models.

How it is built

The architecture, the fusion design, the gold set, and the eval methodology are mine. I work AI-first, directing tools like Claude Code to generate and refactor the implementation while I own the retrieval design and the measurement, then read, run, and check what comes back against the numbers.

Status and contact

Tier: Lab. A working, measured retrieval system, held at lab status on purpose because it runs over a private corpus. What ships here is a sanitized, offline-runnable extract of the design and its eval.

Part of a portfolio of production and lab AI systems. More at github.com/janvrsinsky.

• LinkedIn: linkedin.com/in/janvrsinsky