whensaid
Enables indexing and searching transcripts of YouTube videos, playlists, and channels, using existing captions when available and returning timestamped links that open the video at the matched passage.
This server does not currently have a deployable release on Glama. Follow the repository's setup instructions to install and configure it for your MCP client, including any required credentials and local dependencies.
whensaid
Search what was said in your local video and audio files, from your AI agent.
Point it at folders of recordings (talks, courses, meetings, interviews, podcasts) or at a YouTube playlist.
Your agent (Claude Code, Cursor, ...) calls search("when did they talk about X") and gets back the file, the exact
timestamp and the passage; for web videos, a link that opens the video at that moment. Everything runs on your
machine, and nothing is uploaded.
Status: pre-alpha. Tested on Linux (CI also runs on macOS). Runs on an NVIDIA GPU or on CPU. English only.
A worked example

The recording above is real whensaid output (rendered by scripts/make_demo.py), run in a terminal rather than from an agent session. Text version:
The index here is 198 public-domain recordings (65 hours of audiobooks and old radio). The question is paraphrased, not a quote:
$ whensaid search "a mouse tribe claiming rights over everyone else" -n 1
strong the_consolation_of_philosophy_..._boethius_64kb.mp3 0:01:38 and over whom ye exercise your fancied
lordship suppose now that in the mouse tribe there should rise up one claiming rights and powers for himself above the rest ...Through MCP the agent gets the same thing as JSON (file, path, start, at, text, match, relevance, and
link for web videos) and can call get_transcript to read around the hit. About half a second per search on CPU.
Related MCP server: vidtheque
Use it
pip install "whensaid[url]" # add cuda on an NVIDIA GPU: "whensaid[cuda,url]" (or: uv tool install "whensaid[url]")
whensaid doctor # checks the GPU, models, yt-dlp and the index, says how to fix problems
whensaid add ~/Videos/talks # register a folder of recordings
whensaid add-url https://www.youtube.com/playlist?list=... # or a video, playlist or channel
whensaid index # transcribe and index what is new (models download on first use)
whensaid index --watch # or keep running and pick up new files every 30 s
whensaid index --subtitles # use a .srt/.vtt next to a recording instead of transcribing it
whensaid remove ~/Videos/old # stop watching a folder, playlist or video and drop it from the index
whensaid statusThe models (about 1.6 GB for the default set) are downloaded the first time they are needed. whensaid models pull
fetches them ahead of time, and WHENSAID_OFFLINE=1 forbids any download.
Then give your agent the MCP server (stdio):
claude mcp add whensaid -- whensaid serveTools the agent gets (all read-only): search, get_transcript, list_sources, index_status.
Indexing is only ever done by the whensaid index command, never by the agent.
How good is it
Measured on the 65-hour public-domain index above, with questions written as paraphrases (the question sets are in tests/, the scripts in scripts/):
The right passage is the top result for 83-97% of questions and in the top 5 for 96-100%, depending on the question set.
Off-topic questions: 33 of 40 return nothing, and 6 of the other 7 come back marked
weak.Against an agent that greps plain transcript files (
whensaid export), with Claude Sonnet agents answering 36 questions: grep got 33 right and whensaid 31. That is a tie within noise for one run each, so whensaid is not more accurate than grep at this size. What it adds is speed (an agent finished a batch of 8 questions in about a minute instead of about nine), no export step, timestamp links for web videos, and a library that persists between sessions. One run per arm, questions written by someone who had read the transcripts. An earlier run looked much worse (25 of 36) only because the CLI cut each passage to 200 characters; that is fixed.Not measured: word error rate against human transcripts, noisy or multi-speaker recordings beyond old radio, and any hardware other than one Linux laptop (12 cores, RTX 3050).
Limits
Python 3.11 to 3.14. On 3.15
pip installfails for now because onnxruntime has no wheels for it yet.English only. Videos in other languages, and recordings the model is unsure of, are skipped (see below).
Linux is the tested platform; macOS runs in CI but nobody has used it daily. Windows is untried.
Speech only. It does not look at the picture.
Transcription is the slow part: about 14x realtime on the GPU it was tested on, about 6x on 12 CPU cores with the default model.
WHENSAID_WHISPER_MODEL=small.entrades accuracy for speed.Web videos are for videos you are allowed to download; each site's terms still apply and are your responsibility.
Details
whensaid index also drops files that were deleted or moved (unless their folder is missing or holds no media at all,
which looks like an unplugged drive). With --watch, files are indexed once they have stopped changing for 15 s, and
a file that failed is retried only after it changes.
Web videos use their captions when there are any (seconds, no GPU) and are transcribed from the audio otherwise. Playlists and channels are checked for new videos at most once an hour.
Things that will never work are skipped once and not asked for again: live streams, private or removed videos, videos
that are not in English, and recordings whose transcript the model itself was unsure of (usually another language, or
music). whensaid status lists them with the reason; whensaid index --retry-skipped tries them again, and
WHENSAID_MIN_CONFIDENCE=0 turns the transcript check off.
Set WHENSAID_HOME to keep the index somewhere other than ~/.whensaid.
How it works
ffmpeg-free decode (PyAV) -> faster-whisper with VAD and word timestamps -> ~45 s overlapping
chunks -> bge-small embeddings -> one SQLite file (FTS5 keyword index + vectors).
Search fuses keyword and vector ranks (reciprocal rank fusion), then a small cross-encoder reranks the top 20
(WHENSAID_RERANK=0 skips it). Passages it scores as unrelated are dropped, so off-topic questions usually return
nothing, and borderline hits come back marked weak (WHENSAID_MIN_RELEVANCE tunes this, off disables).
Develop
uv pip install -e ".[dev]" && pytest
WHENSAID_HOME=<index with tests/data indexed> python scripts/eval.py --verboseTest media is public domain; see tests/data/README.md.
License: MIT
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
This server cannot be deployed
Maintenance
Related MCP Connectors
VideoDB lets an AI agent watch, search and clip video.
Transcripts, summaries, chapters and timestamped search for video links, for AI agents.
Search speech in podcasts, government meetings, and your own audio: speakers, entities, timestamps.
Turn any video into context for AI agents: transcripts, metadata, chapters, timestamps.
13
Related MCP Servers
- AlicenseAqualityCmaintenanceLet AI agents watch videos: local transcripts, speaker labels, scenes, chapters and exact-moment search from any video URL or file. Fully local, no API keys.43AGPL 3.0
- AlicenseNot gradedqualityBmaintenanceEnables search across videos you've watched via transcripts, on-screen text, and frames, citing exact timestamps. Point it at videos, channels, or playlists; it indexes everything locally and answers queries with deep links to the exact second.3MIT
- AlicenseNot gradedqualityDmaintenanceEnables LLM agents to process local videos into timestamped, citable text documents and then query them through tools for listing videos, retrieving transcripts, and fetching specific segments, all fully offline.21 PyPIMIT
- AlicenseAqualityBmaintenanceEnables AI agents to analyze videos by producing timestamped transcripts, keyframe contact sheets, and metadata reports from local files or online URLs, fully offline and without API keys.6MIT