video-context

Turn a video into context an AI agent can use — on your machine.

A recorded meeting, a training session or a screencast becomes something an agent can study: "watch this video and tell me how to do the task".

• Timestamped transcript with whisper.cpp.

• Key frames at every scene change (and at least one a minute) with ffmpeg; near-identical frames are dropped.

• Text on screen of each frame with tesseract (optional): URLs, code and field names that are shown but never said.

• An MCP server so an agent (Claude Code or any MCP client) can process a video, read and search the transcript and the screens, and look at the frames as images.

The video never leaves your computer, and no API key is needed: the agent that reads the results does the understanding.

Quick start

# 1. Requirements (macOS: brew install ffmpeg whisper-cpp tesseract)
sudo apt install ffmpeg tesseract-ocr        # and whisper.cpp's whisper-cli, see below

# 2. Process a video
npx -p @lalohao/video-context video-context ./meeting.mp4 --language en

# 3. Or let your agent do it
claude mcp add video-context -- npx -y -p @lalohao/video-context video-context-mcp

"meeting" — 42:10, 611 segments, 118 frames
Transcript: ~/.cache/video-context/videos/3f9c…/transcript.md
Frames:     ~/.cache/video-context/videos/3f9c…/frames
Video id:   3f9c…
Dropped 9 near-duplicate frames.

Then ask Claude, for example:

Process ~/Downloads/onboarding.mp4 and write the setup steps it explains as a checklist, with the minute of each step. Show me the screen where they configure the webhook.

See examples/ for a full walkthrough on a public video, and the docs for the MCP tools and how it works.

Related MCP server: video-to-llm

Requirements

The whisper model is downloaded on first use (465 MB for small) into ~/.cache/video-context/models. Without tesseract, everything works except the text on screen.

Command line

video-context <video> [options]

  -l, --language <code>  Spoken language, e.g. en, es (default: auto)
  -m, --model <name>     tiny, base, small, medium, large-v3-turbo, large-v3 (default: small)
  -o, --out <dir>        Also copy the results to this folder
      --no-ocr           Do not read the text on screen
      --keep-duplicates  Keep near-identical frames
      --force            Process again even if it was processed before

Any file ffmpeg reads works: .mp4, .mkv, .webm, .mov, or audio only (.mp3, .m4a; then there are no frames). Each file is processed once and cached by path, size and modification time.

MCP server

Claude Code: claude mcp add video-context -- npx -y -p @lalohao/video-context video-context-mcp. Other clients and example prompts: docs/mcp.md.

Environment variables

Troubleshooting

• Missing: whisper.cpp's whisper-cli — install whisper.cpp, or set VIDEO_CONTEXT_WHISPER to its binary (older builds call it main).

• Wrong language or gibberish transcript — pass --language; try a bigger model (--model medium).

• Too many or too few frames — very static videos give few frames (one a minute at least); --keep-duplicates keeps every sampled frame.

• Text on screen is noisy — OCR on video frames is approximate; it is meant to find the moment, then the agent looks at the frame itself with get_frames. Install the language data of the video (tesseract-ocr-<lang>) for better results.

• It takes a while — whisper is CPU-bound: an hour of video takes a few minutes with small on a recent machine; tiny/base are faster.

Privacy

Video and audio are processed locally. Only what an agent reads through the tools — transcript text, frame images and their text — reaches its model.

Development

pnpm install
pnpm check                                            # typecheck + unit tests
VIDEO_CONTEXT_MODEL=/path/ggml-small.bin pnpm test    # also the end-to-end test

License

MIT