videoctx

Index a video once. Query it cheaply, forever, with any LLM.

videoctx ("video context") turns any video into an open, portable Video Context index (.vctx, see SPEC.md) using only local, model-free tools: ffmpeg keyframes, PySceneDetect shot boundaries, faster-whisper transcripts, and optional one-line visual captions from any cheap VLM. An agent answers most questions from the index's text layer for fractions of a cent, and pulls individual frames on demand only when a question needs pixels.

videoctx index lecture.mp4          # one-time, local, free
videoctx show  lecture.vctx         # the text view the model reads
videoctx ask   lecture.vctx "What diagram is shown when she explains backprop?"

Why

Native video ingestion re-tokenizes the whole video on every question: an hour of footage is ~250-350k tokens, every single time. Proprietary video platforms index into closed representations behind per-minute pricing. videoctx is the third option: an open on-disk format any model can consume, produced by free tools, where the marginal cost of a follow-up question on an hour-long video is about $0.02.

Related MCP server: video-to-llm

Benchmark results (Video-MME, Oct 2026)

Full protocol and honest cost accounting (including prompt-cache write premiums) in paper/draft.md. Query model: Claude Opus 4.8.

Short videos, held-out test half (n=150):

Identical 30-question subset, all methods:

Long videos, 30-60 minutes (n=30):

One-time index cost: ~$0.10 per hour of video (captions), pennies otherwise.

Install

git clone <repo> && cd videoctx
uv venv && uv pip install -e .

Requires ffmpeg on PATH. videoctx index and the MCP server's core tools run fully locally with no API key. videoctx ask uses the Anthropic API (ANTHROPIC_API_KEY or an ant auth login profile).

MCP server: let any agent read videos

claude mcp add videoctx -- $(pwd)/.venv/bin/videoctx-mcp

The calling agent's own model reads the text layer and frames directly; the server makes no model calls and needs no key. Tools: video_overview (cheap text layer), survey_video (one thumbnail per shot), get_frames(path, t0, t1, detail), transcript_search, index_video. The server's instructions carry the evidence rule: visual answers must cite fetched frames, never general knowledge.

Reproduce the benchmark

uv pip install -e ".[eval]"
python eval/prepare_videomme.py --duration short --videos 100 --out eval/data
python eval/split_dev_test.py eval/data/videomme_short.jsonl
python eval/run.py eval/data/videomme_short_test.jsonl --methods videoctx,naive

eval/methods.py documents each baseline. Cost accounting counts cache writes at their real premium (1.25x / 2x by TTL); see the paper's Section 7.3 for the accounting traps this avoids.

Format

The .vctx directory is plain JSON + JPEG: manifest.json (provenance, parameters, source hash), index.json (shots, keyframes, transcript, captions), frames/. Consumers must ignore unknown keys; enrichment layers (OCR, embeddings) are reserved in SPEC.md v0.1.

Status and roadmap

Working and benchmarked: indexing, query loop, MCP server, eval harness. Next: medium-duration subset, Pro-tier native baseline, cross-model portability runs (same indexes via non-Claude models), OCR and per-shot embedding layers, spec v1.0.

License

Apache-2.0. Copyright 2026 Fred Dunyo.