vidlayer

vidlayer turns a video into timestamped keyframes and a transcript, so that models which only accept text and images can still work with it. The usual workaround is pasting in a transcript, which loses everything that only appears on screen, like slides, code, and demos. You can run vidlayer as an MCP server and let the model request frames itself, or send the output directly to Claude, OpenAI-compatible APIs, or Gemini.

Example

This is the storyboard open_video returns for a one-minute sample talk.

The model also gets the transcript, lined up with the frames.

<video source="talk.mp4" duration="01:04.0" resolution="1920x1080" fps="10.0">
[00:00.5] keyframe (start):        [image]
Transcript:
[00:01.0] Today I'll show how we cut our p99 latency in half.
[00:08.4] keyframe (scene-change): [image]
Transcript:
[00:09.0] Here's where we started. The median was fine, but the tail was nearly a second.
[00:16.4] keyframe (scene-change): [image]
Transcript:
[00:17.0] The flame graph points at one call: we were blocked waiting on a connection.
...

Related MCP server: llm-video-mcp

Install

You will need uv on your PATH and ffmpeg installed (brew install ffmpeg on a Mac). In Claude Code, run the command below.

/plugin install vidlayer --marketplace riceelijah/vid-layer

On a version of Claude Code older than 2.1.275, add the marketplace first.

/plugin marketplace add riceelijah/vid-layer
/plugin install vidlayer@vid-layer

The plugin includes the MCP server and a watch-video skill. Once it's installed, you can ask about a video by giving its path.

What's the UI bug in ~/Videos/demo.mp4 around the 3-minute mark?

MCP tools

A model usually starts with open_video and then requests frames for the parts it needs, which keeps image tokens focused on the moments that matter.

How it works

flowchart LR
    V[Video] --> P[Probe with ffprobe]
    P --> S[Detect scene changes]
    S --> B[Plan frame budget]
    B --> D[Drop near-duplicates]
    P --> T["Find transcript<br/>file, sidecar, embedded, Whisper"]
    D --> DOC[VideoDocument]
    T --> DOC
    DOC --> M[MCP tools]
    DOC --> A["Claude, OpenAI, Gemini<br/>content blocks"]
    DOC --> F[Markdown + JPEG bundle]

ffmpeg's scene detector runs on a downscaled copy of the video to find cuts and slide changes. vidlayer keeps the strongest ones that fit the frame budget and fills long gaps with evenly spaced frames. Any interval frame or weak cut that looks almost the same as the previous frame is dropped, which means that a static screen doesn't use up the budget.

The transcript comes from the first source it finds, which is a file passed with --transcript, a sidecar like video.srt or video.en.vtt, an embedded subtitle track, or local Whisper.

Other ways to use it

pip install "vidlayer[mcp] @ git+https://github.com/riceelijah/vid-layer"

For Claude Desktop, add this to claude_desktop_config.json.

{ "mcpServers": { "vidlayer": { "command": "vidlayer", "args": ["mcp"] } } }

vidlayer build demo.mp4 -o demo.vidlayer

This writes video.md and a folder of JPEGs, which any tool that reads local files can use.

from vidlayer import VideoDocument, to_anthropic, to_openai, to_gemini

doc = VideoDocument.build("talk.mp4", max_frames=32)

# Claude
client.messages.create(model="claude-opus-5-5", max_tokens=16000, messages=[
    {"role": "user", "content": to_anthropic(doc) + [{"type": "text", "text": "Summarize the slides."}]}
])

# OpenAI-compatible APIs
{"role": "user", "content": to_openai(doc) + [{"type": "text", "text": "..."}]}

# Gemini (REST `parts`)
{"role": "user", "parts": to_gemini(doc) + [{"text": "..."}]}

vidlayer ask demo.mp4 "What command gets run at the end?"   # needs ANTHROPIC_API_KEY
vidlayer render demo.mp4 --format openai > content.json
vidlayer build "https://youtube.com/watch?v=..."            # needs vidlayer[url]

Options include --max-frames, --max-dim, --scene-threshold, --transcript subs.vtt, and --no-asr.

pip install -e ".[mcp]"   # MCP server
pip install -e ".[all]"   # also Whisper, yt-dlp, and the Anthropic SDK

Token cost

Claude charges roughly width × height / 750 tokens per image, and vidlayer build prints an estimate for each video.

Roadmap

• OCR on frames, for searching on-screen text, text-only models, and catching small slide edits

• Speaker diarization

• Audio events other than speech (music, laughter, alarms)

• Per-scene captions from a cheap vision model

• More frames where the speaker refers to the screen

• On-disk cache for the MCP server

• get_clip tool for models that accept video

Development

pip install -e ".[dev,mcp]"
pytest

The tests generate their own video with ffmpeg, so there are no fixture files.

License

MIT