The Librarian

Agentic RAG over a private podcast archive, measured before it was trusted.

Portfolio exhibit. This is a sanitized public extract of a private system in daily use. The architecture and method are real; the data and identifiers are stand-ins, and the section "What ships here, and what is sanitized" below lists which is which.

What it does

The Librarian answers questions over a 55-episode podcast archive: research notes plus roughly 19 hours of audio, transcribed with timestamps by Whisper and chunked into about 2,200 addressable passages. Each question runs through two retrieval branches, lexical and semantic, fused into one ranking that cites the exact passage and minute. A 122-query gold set scores recall@k and MRR across four strategies, offline, with one command.

▶ Watch the demos

Related MCP server: confluence

The eval is the signal

Retrieval quality is a measurable problem, so it gets measured. eval/gold_set.json holds 122 paraphrase queries, two per episode, each written against one target chunk and worded to avoid that chunk's surface tokens, so the eval scores paraphrase retrieval rather than string matching. eval/run.py scores recall@1, recall@3, recall@5, and MRR for four retrieval strategies over the 545-chunk sample corpus and prints the table live. Nothing is hard-coded.

It runs offline with nothing but the Python standard library:

python eval/run.py

Output from the default offline run in this repo:

corpus: 545 chunks   gold queries: 122
dense backend: fallback-char-ngram
(offline fallback dense vectors; set USE_ST=1 for real embeddings)

Retrieval mode         | R@1    | R@3    | R@5    | MRR
-----------------------+--------+--------+--------+-------
BM25, raw tokens       | 0.47   | 0.60   | 0.64   | 0.54
BM25, normalized       | 0.50   | 0.66   | 0.71   | 0.58
Dense, semantic        | 0.25   | 0.34   | 0.42   | 0.31
Hybrid, RRF            | 0.42   | 0.57   | 0.66   | 0.51

best MRR: BM25, normalized (0.58)

Those numbers are gated in CI: every push recomputes the table and fails the build if any tracked metric falls below its floor, if fusion stops improving on its weaker branch, or if the table above drifts from a live run. The floors sit four to five points under the measured values, about six flipped queries out of 122, so ordinary noise passes and a real regression fails.

An earlier version of this eval ran 12 queries over 15 chunks. On a corpus that small every strategy lands near 1.00 and the table mostly proves the wiring, so the corpus was scaled until the numbers could disagree. At 545 chunks they do. Normalized BM25 recovers inflected matches that raw tokens miss (recall@3 0.66 against 0.60). The offline fallback vectors collapse at this scale (MRR 0.31), as surface n-grams should on paraphrase queries. And fusing that weak branch in drags the hybrid below plain normalized BM25 (MRR 0.51 against 0.58). A third of the queries put nothing relevant in the lexical top three, which is what the paraphrase wording is there to force.

With USE_ST=1 the dense branch swaps the fallback for real multilingual-e5 embeddings. Same corpus, same gold set, measured by hand (the model is a download, so CI stays on the offline path):

Retrieval mode         | R@1    | R@3    | R@5    | MRR
-----------------------+--------+--------+--------+-------
BM25, raw tokens       | 0.47   | 0.60   | 0.64   | 0.54
BM25, normalized       | 0.50   | 0.66   | 0.71   | 0.58
Dense, semantic        | 0.73   | 0.89   | 0.94   | 0.81
Hybrid, RRF            | 0.61   | 0.81   | 0.92   | 0.72

The same lesson from the other side: real embeddings win the table outright (MRR 0.81), and RRF fusion with the much weaker lexical branch gives part of that lead back (0.72). At this corpus size, whichever branch is stronger, fusing in a much weaker partner costs rank quality. The 15-chunk eval showed hybrid on top and hid all of this; scaling the corpus is what gave the eval the resolution to catch it, and catching it is what an eval is for.

The same eval, running on Azure

The dense branch above is the project's known weak point: offline it is a char-n-gram stand-in, and even the real local model is a download CI will not carry. So the retrieval stack was ported to managed infrastructure and measured against the same bar: Azure OpenAI for embeddings, Azure AI Search for the index and for server-side RRF fusion.

azure_slice/ holds the port and exposes search(mode, query, k) with the same signature as the local Retrievers, so eval/run_azure.py reuses the local metric functions rather than reimplementing them. Same 545 chunks, same 122 paraphrase queries, same k. Two choices keep it honest: only text is searchable on the Azure index, exactly as the local BM25 index is built over chunk text alone; and the Azure index uses the Czech analyzer, which is the counterpart of the local normalized branch rather than the raw one.

Running it takes the two Azure resources, two packages, and six settings read from a gitignored .env.azure. No endpoint, key or resource name ships in this repo:

pip install -r requirements-azure.txt

# .env.azure, values are your own:
#   AZURE_OPENAI_ENDPOINT   AZURE_OPENAI_API_KEY   AZURE_OPENAI_EMBED_DEPLOYMENT
#   AZURE_SEARCH_ENDPOINT   AZURE_SEARCH_API_KEY   AZURE_SEARCH_INDEX

python -m azure_slice.index   # create the index, embed and upload the corpus
python eval/run_azure.py      # score both stacks side by side

Measured by hand on 28 August 2026, for the same reason the USE_ST=1 table above is: this run needs live credentials and a billed endpoint, so CI cannot reproduce it. The table CI gates is the offline one at the top of this README; the seven rows below are a hand-run record, and the local four are reproduced here from the same sweep so both stacks are read off one run.

Retrieval mode             | R@1    | R@3    | R@5    | MRR
---------------------------+--------+--------+--------+-------
local · BM25 raw           | 0.47   | 0.60   | 0.64   | 0.54
local · BM25 normalized    | 0.50   | 0.66   | 0.71   | 0.58
local · dense (fallback)   | 0.25   | 0.34   | 0.42   | 0.31
local · hybrid RRF         | 0.42   | 0.57   | 0.66   | 0.51
azure · BM25 (cs analyzer) | 0.38   | 0.57   | 0.61   | 0.47
azure · vector (AOAI)      | 0.77   | 0.97   | 0.98   | 0.86
azure · hybrid RRF         | 0.62   | 0.88   | 0.98   | 0.76

text-embedding-3-small is the best dense branch this corpus has seen: MRR 0.86 against 0.81 for local multilingual-e5 on the same gold set, and 0.31 for the offline fallback. The architecture did not change between those three rows; the embedding model did.

The fusion lesson replicates for the third time. Azure hybrid (0.76) lands below Azure vector alone (0.86), the same way local hybrid lands below whichever local branch is stronger, with the fallback dense branch and with real e5 alike. Three independent stacks, one behaviour: at this corpus size RRF gives back part of the stronger branch's lead. That is a property of fusing uneven branches, not a property of any one vendor.

One result did not replicate: the cs.microsoft analyzer scores below the repo's own stemmer (0.47 against 0.58). Managed does not automatically mean better on inflected Czech.

The gold set is deliberately paraphrased, which favours semantic retrieval; on names, codes and exact terms the lexical branches would read differently. These numbers rank this gold set and nothing wider.

Cost for the whole exercise, indexing plus a full eval sweep: $0.0006. The search service runs on the free tier; embeddings are billed per token.

What ships here, and what is sanitized

Real: the architecture and the measurement. The two-branch retrieval, the RRF fusion, the gold-set eval, the recall@k and MRR scoring, and the typed MCP surface run end to end in this repo.

Everything below is a deliberate, labeled stand-in, so the pipeline runs offline with zero downloads while the real corpus stays private:

• Corpus. The shipped sample is 545 timestamped chunks across 55 episodes, roughly a quarter of the full corpus size. The passages are synthetic English stand-ins written for this repo: they mirror the archive's shape (episode structure, notes and transcript layers, timestamps, podcast-register topics); none of the real Czech episode content appears, and neither do guest or host names.

• Normalization. The full system uses a real Czech lemmatizer (simplemma); the sample ships a small standard-library suffix stemmer that reproduces the effect that matters for the score, matching inflected forms to a shared stem.

• Embeddings. The full dense branch runs multilingual-e5 locally; the shipped eval defaults to an offline char-n-gram vector fallback, labeled as such at every boundary. USE_ST=1 turns the real model on.

• Not in this repo. Whisper transcription and the grounded answer-generation layer. This repo measures the retrieval beneath them.

Demos

Three short clips, recorded live against the sample corpus, one capability each.

Audio-only knowledge. A fact that exists only in the spoken transcript: how fast electrons actually drift versus how fast the signal travels. Retrieved and cited to the minute.

https://github.com/user-attachments/assets/4ab15dd1-6891-4f4c-b023-9ac0a22b21df

A concept spoken, never written down. The name for the mythic descent into the underworld appears only in the audio; a paraphrased question sharing none of its surface words still lands on the right passage.

https://github.com/user-attachments/assets/02ac8608-871e-482e-bc17-3e5ca339227f

Cross-episode synthesis. An answer spread across more than one episode; retrieval pulls the passages from each, and every claim keeps its own timestamped citation.

https://github.com/user-attachments/assets/a755d3ce-929d-4ae7-bfe5-3a7ca8569e7a

How it works

• Two branches, one fused ranking. BM25 scores lexical overlap, the dense branch semantic similarity; reciprocal rank fusion (constant c=60, from Cormack et al.) combines their ranks, since the raw scores share no scale.

• One place defines every branch. retrieval/pipeline.py builds bm25_raw, bm25_stem, dense, and hybrid once; the eval and the MCP server can never drift apart on what those modes mean.

• Grounded, addressable answers. Every chunk carries its episode, source layer (notes vs transcript), and start time; each result returns a citation tag like Ep. ep03 @ 08:03.

flowchart TB
    NOTES["Research notes, 55 episodes"] --> CHUNK
    AUDIO["Audio, 19 h, Whisper timestamped transcripts"] --> CHUNK["Timestamped chunks<br/>~2200 full, 545 in this sample"]
    CHUNK --> BM25["BM25 lexical<br/>raw + normalized"]
    CHUNK --> DENSE["Dense embeddings<br/>multilingual-e5 or offline fallback"]
    BM25 --> RRF["Reciprocal rank fusion, c=60"]
    DENSE --> RRF
    RRF --> MCP["MCP server, 3 typed tools<br/>ranked, cited passages"]
    RRF --> EVAL["Eval, recall@k and MRR<br/>gold set, 122 queries"]
    MCP --> ANS["Grounded answer, cited to the minute<br/>answer layer not in this repo"]

The MCP surface

A FastMCP server exposes the exact pipeline the eval measures through three typed tools; the numbers above describe what an agent receives.

python mcp_server.py            # stdio transport
MCP_HTTP=1 python mcp_server.py # streamable HTTP on 127.0.0.1:8765

• search_podcast(query, k=5, mode="hybrid") returns ranked, timestamped, cited passages. mode is one of bm25_raw, bm25_stem, dense, hybrid.

• list_episodes() returns the episode catalog with per-episode chunk counts.

• episode_detail(episode_id) returns one episode's chunks in timestamp order.

Stack

The default python eval/run.py install is empty: standard library only. mcp>=1.2.0 runs the server; sentence-transformers, simplemma, and faster-whisper are optional, for the real models. requirements-azure.txt is separate again and only the Azure slice needs it, which is why CI installs neither.

Status and contact

PRODUCTION EXTRACT. A sanitized public cut of a private system in real use. I direct AI coding tools to build it; the architecture, the fusion design, the gold set, and the eval methodology are mine.

More of the portfolio at github.com/janvrsinsky.

• LinkedIn: linkedin.com/in/janvrsinsky

Topics