video-context

Turn a video into context an AI agent can use — on your machine.

A recorded meeting, a training session or a screencast becomes something an agent can study: "watch this video and tell me how to do the task".

• Timestamped transcript with whisper.cpp, or mlx-whisper on the GPU of Apple silicon Macs.

• Key frames at every scene change (and at least one a minute) with ffmpeg; near-identical frames are dropped.

• Text on screen of each frame with tesseract (optional): URLs, code and field names that are shown but never said.

• An MCP server so an agent (Claude Code or any MCP client) can process a video, read and search the transcript and the screens, and look at the frames as images.

The video never leaves your computer, and no API key is needed: the agent that reads the results does the understanding.

Quick start

1. Install the tools — ffmpeg and a whisper engine (tesseract is optional):

Other systems, GPUs and troubleshooting: docs/install.md.

2. Check the setup

npx -p @lalohao/video-context video-context --doctor

3. Process a video, or let your agent do it:

npx -p @lalohao/video-context video-context ./meeting.mp4 --language en
claude mcp add video-context -- npx -y -p @lalohao/video-context video-context-mcp

"meeting" — 42:10, 611 segments, 118 frames
Transcript: ~/.cache/video-context/videos/3f9c…/transcript.md
Frames:     ~/.cache/video-context/videos/3f9c…/frames
Video id:   3f9c…
Dropped 9 near-duplicate frames.

Then ask Claude, for example:

Process ~/Downloads/onboarding.mp4 and write the setup steps it explains as a checklist, with the minute of each step. Show me the screen where they configure the webhook.

See examples/ for a full walkthrough on a public video, and the docs for the MCP tools and how it works.

Related MCP server: video-to-llm

Requirements

When both engines are installed, mlx-whisper is used (--engine picks one). The whisper model is downloaded on first use (465 MB for small): into ~/.cache/video-context/models for whisper.cpp, into the Hugging Face cache for mlx-whisper. Without tesseract, everything works except the text on screen. Step-by-step for each system: docs/install.md.

Command line

video-context <video> [options]
video-context --doctor

  -l, --language <code>  Spoken language, e.g. en, es (default: auto)
  -m, --model <name>     tiny, base, small, medium, large-v3-turbo, large-v3 (default: small)
  -e, --engine <name>    whisper.cpp or mlx (default: mlx when installed, else whisper.cpp)
  -o, --out <dir>        Also copy the results to this folder
  -n, --name <name>      A name to call it by (instead of the file name)
  -f, --folder <folder>  File it in a folder, e.g. "Posgrado/Clases"
  -t, --tag <tag>        Tag it (repeat for several tags)
      --no-ocr           Do not read the text on screen
      --keep-duplicates  Keep near-identical frames
      --force            Process again even if it was processed before
      --doctor           Check the setup: what is installed, and how to install the rest

Any file ffmpeg reads works: .mp4, .mkv, .webm, .mov, or audio only (.mp3, .m4a; then there are no frames). Each file is processed once and cached by path, size and modification time.

MCP server

Every tool takes a video by its name — or words of its name, folder or tags — as well as by its id: "in my Posgrado videos, where did they explain OSPF?".

Claude Code: claude mcp add video-context -- npx -y -p @lalohao/video-context video-context-mcp. Other clients and example prompts: docs/mcp.md.

Web page

npx -p @lalohao/video-context video-context-ui --open

A page on http://localhost:4317 to watch a processed video next to its transcript: the current line follows the player, a click on a line or a key frame seeks there, and the search finds words said or shown on screen, in this video or in all of them. + New video adds files (dropped on the page or by path) to a processing queue, with the language, whisper model and engine you pick; the queue shows each step's progress, and can cancel, retry or reorder.

• Ask Claude through the claude CLI (your own login; pick the model at the top), about this video or about all your videos at once ("in which meeting did they decide the deadline?"). Claude gets the transcript and this package's MCP tools, so it can search and look at the screen — the key frames, and the exact screen at any moment, to read code being typed; what it looked at shows in the answer, and every time it cites ([3:07], [V2 3:07] for another video) is a link to that moment. Examples: docs/claude.md.

• Chapters: Claude splits the video where the topic changes; they show on the bar under the player, in the transcript and in their own tab.

• Clips: mark a start and an end ([ and ], a range in an answer, or selected transcript lines) and download them as MP4, or as a GIF up to 30 s.

• Notes: write one at the current time (N), bookmark it (B), or save an answer from the chat; they are kept next to the video (notes.json) and exported to Markdown with the chapters and the conversation.

• Queue and logs: videos are processed one after another, each step with its progress; a video stopped halfway (even by a crash) goes on from the steps it finished. The logs say what failed and how to fix it.

• Export: the video as a ZIP file with its transcript, frames, chapters, notes and an index.html that opens without a server; dropped on another video-context, it is ready without processing it again.

The page listens only on 127.0.0.1. A tour with screenshots of each part: docs/ui.md.

Who said what

A recorder can leave a .json file with the same name next to the video (daily.webm + daily.json). When it says who was speaking when, every line of the transcript carries the speaker's name, and search_transcript finds a name together with the words ("Ana pagos"). Its title replaces the file name.

{
  "schema": "video-context-recording/1",
  "title": "Daily",
  "recordingStartedAt": "2026-10-22T16:00:00.000Z",
  "speakers": [
    { "name": "Ana", "start": 12.4, "end": 18.9, "source": "captions" },
    { "name": "Carlos", "start": 19.1, "end": 31.0, "source": "captions" }
  ]
}

Times are seconds from the start of the video. Each transcript segment gets the speaker whose turns overlap it the most (or the nearest turn within 2 s). source is captions (the meeting's live captions) or indicator (a participant tile's audio-level animation, less reliable: used only when there are no caption turns). A video processed before its .json existed needs --force (in the web page, Process again in the queue). In the web page, drop the video and its .json together.

Environment variables

Troubleshooting

• Missing: … — run video-context --doctor: it says what is installed and how to install the rest on your system. A tool installed somewhere else: set its variable (VIDEO_CONTEXT_WHISPER to whisper.cpp's binary, which older builds call main).

• It works in the terminal but the MCP server says a tool is missing — desktop apps may not see your shell's PATH: give the server the full paths.

• Wrong language or gibberish transcript — pass --language; try a bigger model (--model medium).

• Too many or too few frames — very static videos give few frames (one a minute at least); --keep-duplicates keeps every sampled frame.

• Text on screen is noisy — OCR on video frames is approximate; it is meant to find the moment, then the agent looks at the frame itself with get_frames. Install the language data of the video (tesseract-ocr-<lang>) for better results.

• It takes a while — an hour of video takes a few minutes with small on a recent machine; tiny/base are faster. On a Mac with Apple silicon, mlx-whisper (on the GPU) is usually faster than whisper.cpp; on Linux, build whisper.cpp with CUDA for an NVIDIA GPU.

• The first video with mlx-whisper seems stuck at 0 % — it is downloading the model from Hugging Face (once).

• A browser's recording plays but can't be seeked (a meeting or screen recorder made it with MediaRecorder: no duration, no index) — video-context makes a seekable copy when it processes one (playable.webm in its folder, no re-encoding), and the page, clips and exports use it; the original file is not changed. A video processed before this gets its copy the first time it is opened in the page.

• sh: video-context-ui: not found (or video-context, video-context-mcp) — npx -p @lalohao/video-context … was run inside a clone of this repository: npx takes the folder's own package for the published one, and its commands are not installed there. Run it from any other folder, or, in the clone, use the development commands.

Privacy

Video and audio are processed locally. Only what an agent reads through the tools — transcript text, frame images and their text — reaches its model.

Development

pnpm install
pnpm check                                            # typecheck + unit tests
VIDEO_CONTEXT_MODEL=/path/ggml-small.bin pnpm test    # also the end-to-end test (whisper.cpp)
VIDEO_CONTEXT_ENGINE=mlx pnpm test                    # the end-to-end test with mlx-whisper

In a clone, run the commands from the build (npx -p @lalohao/video-context … does not work inside the repository, see Troubleshooting):

pnpm ui                                               # build, then the web page (video-context-ui)
node dist/cli.js ./meeting.mp4 --language en          # video-context
claude mcp add video-context-dev -- node "$PWD/dist/mcp.js"   # the MCP server from this clone

License

MIT