Distributed Video QA MCP

Give your MCP client a video link. It downloads the video with sound, transcribes it, indexes every scene, and answers questions with timestamps and the actual frames. On a Mac, all of it runs locally, with no API keys.

v0.3.0 · 19 typed tools · 5 resources · 5 prompts · local MLX or hosted backends · Python 3.11+ · MIT

Quick start · Backends · Verification · Interface · Tool contract · How it works · Operations · Changelog

Try asking your MCP client:

Index https://www.youtube.com/watch?v=… and list every claim the speaker makes about pricing, with timestamps.

Find the moment the architecture diagram is on screen and read me its labels.

Ingest the first 20 videos of this playlist and tell me where they disagree about the roadmap.

Download this link at 1080p with audio into ~/Movies.

What it does

Related MCP server: mcptube-vision

Quick start

On Apple Silicon (fully local, no keys)

Requires FFmpeg on PATH.

git clone https://github.com/arthurcolle/distributed-video-qa-mcp.git
cd distributed-video-qa-mcp
uv venv && uv pip install -e ".[mlx]"                   # MLX Whisper + MLX jina-embeddings-v4
.venv/bin/distributed-video-qa-mcp fetch-models         # one-time 3.9 GB download of the embedding model
.venv/bin/distributed-video-qa-mcp doctor               # prints the backends; exits non-zero if not ready

The model files are pinned to exact Hugging Face revisions and cached in ~/.cache/huggingface. While loaded, the embedding model uses about 5 GB of unified memory.

Anywhere else

uv pip install -e ".[faster-whisper]"     # local Whisper on Linux/CUDA/CPU, or set GROQ_API_KEY instead
export JINA_API_KEY=…                     # hosted jina-embeddings-v4

Add .[jev] and set TYPESAFE_API_KEY to enable verification. The install includes a Deno runtime for YouTube's signature challenge; without one, YouTube offers only low qualities.

Claude Code

claude mcp add video-qa -- "$PWD/.venv/bin/distributed-video-qa-mcp"     # local Mac setup: no env needed

Any MCP host (stdio)

{
  "mcpServers": {
    "video-qa": {
      "command": "/absolute/path/distributed-video-qa-mcp/.venv/bin/distributed-video-qa-mcp",
      "env": { "TYPESAFE_API_KEY": "… optional, for verify" }
    }
  }
}

Shared server (streamable HTTP)

DVQA_HTTP_TOKEN=$(openssl rand -hex 32) distributed-video-qa-mcp serve --http --host 0.0.0.0 --port 4005 \
  --allowed-host video-qa.internal:4005

Clients send Authorization: Bearer <token> to /mcp; /health is open. The server refuses to bind a non-loopback address without a token. See operations for TLS, process supervision and backups.

Backends

DVQA_EMBED_BACKEND=auto (the default) picks local MLX on Apple Silicon when the mlx extra is installed, and otherwise the Jina API when JINA_API_KEY is set. Transcription works the same way through DVQA_TRANSCRIBE_BACKEND.

Because the vectors match, both backends store them under the model name jina-embeddings-v4. An index built with one is searched with the other, and switching backends re-embeds nothing.

About the MLX port. Jina's published MLX code has three behaviours that are wrong for a long-running server. It caches results keyed on Python object ids, so a later input can silently get an earlier input's vector. It frees the vision weights after the first image. Its compiled text path ignores padding in some batches. The server loads the published weights through the port's model classes but bypasses those paths. embed_mlx.py documents the details, and tests/test_embed_mlx.py pins each failure: distinct images must get distinct vectors, a repeated image the same vector, and batches must match single inputs.

Verification with Jev

Similarity measures what is near a question, not whether it answers it. On ten test questions against one video, the best transcript similarity was 0.51–0.76 for questions the video answers and 0.42–0.55 for questions it doesn't. The ranges overlap, so no threshold works. Jev 1.14 is TypeSafe's typed decision model: instead of prose, it returns a probability for a yes/no question. Asked "does this evidence contain a direct answer?" for each retrieved moment, it separated the two groups cleanly.

video_ask(question, verify=true) asks Jev about every candidate moment's transcript and every frame. It ranks moments by their strongest support and returns the probabilities on each moment (transcript_answers, each frame's shows, support). It sets evidence.supported, which is true when any moment reaches 0.5; that threshold is this server's policy, not the model's. When nothing is supported, it judges the best frame matches individually as a rescue, because some answers are only on screen. A verified ask costs about 0.7–0.9 s and ~15k input tokens (~1,100 per judgement).

Frame judgements are noisier than transcript judgements on questions that aren't visual: a website screenshot scored 0.70 for "what website does he recommend". Read transcript_answers and shows separately when it matters.

Verification is off by default because it sends transcript excerpts and frames to TypeSafe. Turn it on per call, or for every call with DVQA_VERIFY=jev. It needs the jev extra and TYPESAFE_API_KEY, and it pins jev-1.14.0, failing if the provider answers with a different model.

Interface

Every tool declares a typed result. The JSON Schema is published as the tool's outputSchema, every response is validated against it before it is sent, and it arrives as structuredContent. The full per-argument and per-field reference is docs/tool-contract.md. It is generated from the code, and a test fails if it goes stale.

Anatomy of an answer

video_ask(question="Why did he lose money on the Seoul bet?", max_moments=1, frames_per_moment=1) against a 6½-minute video returns the structure below (transcript shortened), followed by a caption and the JPEG of each listed frame:

{
  "question": "Why did he lose money on the Seoul bet?",
  "moments": [{
    "video_id": "cb5590f118d1",
    "span": "3:44-4:48",
    "matched_by": ["said_keyword", "said_semantic", "shown"],
    "transcript": [
      { "at": "3:48", "start": 228.7, "text": "So let me tell you about my Seoul, South Korea bet." },
      { "at": "3:58", "start": 238.6, "text": "But I sized in heavy, $300, it was at 62 cents." },
      { "at": "4:02", "start": 242.1, "text": "The problem? Seoul's temperature crept up all day and hit exactly 13." },
      { "at": "4:09", "start": 249.4, "text": "And under 13 means under, not equal to." }
    ],
    "frames": [{ "at": "4:01", "t": 241.9, "path": "…/frames/00066_000241900.jpg" }]
  }],
  "evidence": { "best_said_similarity": 0.62, "best_shown_similarity": 0.48, "keyword_matches": 9 },
  "guidance": "Answer only from this evidence and cite times as m:ss. Images follow in moment order."
}

evidence lets the model tell a real match from a near miss. On the same index, the off-topic question "What is the capital of Australia?" scored best_said_similarity 0.44 with 0 keyword matches.

Errors

A failed call returns isError: true and text of a fixed format, so the code can be branched on:

Error executing tool video_ingest: path_not_allowed: /private/etc/hosts is outside the allowed roots; set DVQA_ALLOWED_ROOTS to widen them — Add the folder to DVQA_ALLOWED_ROOTS.

There are 32 registered codes in five categories (input, access, source, backend, state). Each one has a hint and a retryable flag, listed in the contract. A test fails if the code raises an unregistered one.

Resources and prompts

How it works

                 ┌─ audio ─ Whisper (on-device, or silence-cut chunks in parallel) ─ passages ─ text vectors ─┐
source ─ probe ──┤                                                                                            ├─ SQLite index
                 └─ frames (parallel FFmpeg segments: interval + scene cuts) ─ dedupe ─ image vectors ────────┘

• Concurrent branches, resumable stages. Speech and frames run at the same time. Each branch records a signature of its inputs (backend, model, dimensions, sampling settings) when it finishes. Re-ingesting skips matching stages, and changing a setting redoes only the stage it affects. Jobs left running by a crash are marked interrupted at startup.

• Retrieval. Transcript segments are grouped into ~30-second passages that overlap by one segment. A query runs dense passage search, BM25 over the passages (SQLite FTS5, porter stemming, stopwords removed), and dense frame search. Results are fused by reciprocal rank (k = 60). The two transcript channels are weighted ½ each, so a strong frame match is not outvoted two to one. Nearby hits merge into moments of at most 75 seconds.

• Index. A single SQLite file in WAL mode holds segments, passages, frames, the FTS table and float32 vectors. Vector search is exact: each vector set is loaded once into a NumPy matrix and scored with one matrix product.

• Rate budget. The Jina client holds a sliding one-minute token budget, seeded from measured per-image costs and corrected from every usage report. A large ingest runs at your rate limit instead of failing on it.

Measured

These numbers are from an Apple M4 Max on the 6:36 video above, plus synthetic data:

Configuration

Settings are environment variables. The defaults suit a laptop, and the operations guide explains how to tune them.

Security

• URLs. Only plain http(s) to public addresses is accepted. Loopback, private, link-local and cloud-metadata targets are refused unless you opt in, as are embedded credentials.

• Paths. Local paths are resolved, symlinks included, and must sit under DVQA_ALLOWED_ROOTS. In HTTP mode nothing outside the state directory is reachable by default.

• Deletion. video_delete removes index rows and the server's own copies. A file you ingested by path is never deleted.

• Process. Subprocesses run without a shell, logs go to stderr, and the index directory is created 0700.

See SECURITY.md for the threat model and how to report a vulnerability. Downloading is subject to each site's terms and to copyright law; use this on media you have the right to copy.

Development

uv pip install -e ".[dev]"
ruff check src tests && ruff format --check src tests
pytest                                                         # 36 tests; the 3 MLX tests skip off Apple Silicon
distributed-video-qa-mcp contract -o docs/tool-contract.md     # after changing any tool or model

The tests build a synthetic video with FFmpeg and use an offline embedder and a scripted transcriber. They drive the whole pipeline through a real MCP client and check every result against its declared model. A fake verifier covers reranking, the frame rescue and the unsupported verdict. They also require a description on every argument and field, a registered code for every error raised, and a contract document that matches the code.

License

MIT. The Distributed Systems name and wordmark are not covered; see LICENSE.