video-deconstructor

Reverse-engineer how any video is made, using the AI model you already have.

Language models can't watch videos inside coding agents. video-deconstructor turns a YouTube URL or a local video into files a multimodal model can read: timestamped storyboard sheets, full-resolution style frames, an audio strip with cut markers, measured pacing statistics and a shot-by-shot A/V script. A bundled skill then guides the model through a three-pass breakdown of structure, visual design, and motion and transitions, ending in a recreation recipe.

It works as an MCP server (GitHub Copilot, Claude Desktop / Claude Code, Cursor, VS Code), a CLI, and an agent skill. It needs no API keys and has no npm dependencies, and it runs on Windows, macOS and Linux. Two optional add-ons are included: local Whisper transcription for videos without captions, and a Gemini pass-through that uses your own API key.

flowchart LR
    A[YouTube URL<br/>or local video] --> B[yt-dlp<br/>download + captions]
    B --> C[ffmpeg<br/>one analysis pass]
    C --> D1[av-timeline.md]
    C --> D2[pacing-metrics.json]
    C --> D3[contact-sheets/*.jpg]
    C --> D4[style-frames/*.jpg]
    C --> D5[audio-strip.png]
    D1 & D2 & D3 & D4 & D5 --> E[Multimodal model<br/>+ 3-pass skill]
    E --> F[How it was made<br/>+ recreation recipe]

What you get

Example (Vox, The school-to-prison pipeline, explained): the storyboard sheet shows the newspaper-clipping style with a highlighter reveal, then a cut to a data chart. The audio strip lines up 33 hard cuts and 2 diamond-wipe transitions against the narration.

Related MCP server: Understand This Video For Me AI Plugin

Quick start

1. Install the prerequisites

yt-dlp needs a JavaScript runtime to download from YouTube. If Deno is not installed, video-deconstructor passes the Node.js it runs on to yt-dlp instead.

2. Get the code and check your setup

Run it straight from GitHub, with no clone or install (npm downloads it on first use):

npx github:shivam098/video-deconstructor doctor
npx github:shivam098/video-deconstructor overview "https://www.youtube.com/watch?v=HoKkasEyDOI"

Or clone it if you want to modify it or run the tests:

git clone https://github.com/shivam098/video-deconstructor.git
cd video-deconstructor
node bin/cli.mjs doctor

doctor runs every candidate binary, skips broken ones and tells you what's missing:

video-deconstructor 0.1.0 on win32-arm64, Node v26.13.0
  OK   yt-dlp 2026.10.26  (...)
  OK   ffmpeg 9.0.2-full_build-www.gyan.dev  (...)
  OK   ffprobe 9.0.2-full_build-www.gyan.dev  (...)
  OK   ffmpeg filters
Ready.

3. Analyze a video

node bin/cli.mjs overview "https://www.youtube.com/watch?v=HoKkasEyDOI"

The school-to-prison pipeline, explained - Vox
Range 00:00.000-03:16.023 | 1920x1080 @ 24 fps
Shots 36 (33 hard cuts, 2 soft transitions) | cuts/min 10.7 | avg shot 5.45s | median 2.25s | low-motion 9%
Captions: 80 lines (uploader subtitles) | silences: 2
Motion hotspots: 16-19s, 43-46s, 47-50s, 68-71s, 182-185s
Output: ./vd-output/youtube-HoKkasEyDOI

Then open the output folder in your agent and ask it to analyze the video, or connect the MCP server (below) and let it do everything.

Use it from an AI agent

As an MCP server

Add to your MCP client configuration. Use absolute paths.

GitHub Copilot CLI / app (~/.copilot/mcp-config.json). The simplest option runs it from GitHub with npx:

{
  "mcpServers": {
    "video-deconstructor": {
      "type": "local",
      "command": "npx",
      "args": ["-y", "github:shivam098/video-deconstructor", "mcp"],
      "tools": ["*"]
    }
  }
}

For a local clone, use "command": "node" and "args": ["/absolute/path/to/video-deconstructor/bin/cli.mjs", "mcp"].

Claude Desktop (claude_desktop_config.json), Cursor (.cursor/mcp.json) and VS Code (.vscode/mcp.json, key servers instead of mcpServers) use the same command / args.

Claude Code:

claude mcp add video-deconstructor -- npx -y github:shivam098/video-deconstructor mcp

Tools:

Every tool returns file paths. Agents that can open local files (Copilot, Claude Code, Cursor) view them directly. For clients that can't, such as Claude Desktop, pass inline_images: N to receive the first N images inline.

As a skill

Copy or link skills/video-production-deconstructor/ into your agent's skills folder (for example ~/.copilot/skills/ or ~/.claude/skills/). Then ask things like:

Break down how this Kurzgesagt video was animated and give me a recipe to recreate the opening in After Effects: https://www.youtube.com/watch?v=fEjvSbI_PT4

The skill tells the agent to run the overview, read the timeline and metrics, view the sheets and style frames, zoom into the motion hotspots, and report measured facts separately from inferences. See a complete example report.

Optional add-ons

Both add-ons are off by default; the core tool needs neither.

Local transcription with Whisper (free, offline)

Many videos have no captions. Whisper is OpenAI's open-source speech-to-text model. It runs on your machine through whisper.cpp: ffmpeg 8+ builds include it as the whisper filter (Gyan's Windows build does), or you can install whisper-cli (brew install whisper-cpp).

node bin/cli.mjs setup-whisper                    # one-time: downloads base.en (~142 MB)
node bin/cli.mjs setup-whisper --model small      # multilingual; others: tiny(.en), base(.en), small(.en), medium(.en), large-v3-turbo
node bin/cli.mjs transcribe <url|file> [--start T] [--end T] [--language hi] [--translate]
node bin/cli.mjs overview <url|file> --transcribe always   # use Whisper even when captions exist

With a model installed, overview transcribes automatically whenever a video has no captions (--transcribe auto, the default). Transcripts are cached, so later zoom calls reuse them. On the test machine, base.en transcribed 60 s of narration in about 5 s, with output close to the uploader's subtitles.

Gemini native video understanding (your own API key)

The local tools give the model measured cut data and still images. Gemini can watch the video itself, motion and sound included, and write its own analysis. gemini_analyze_video (CLI: gemini) adds that as a second opinion:

export GEMINI_API_KEY=...        # PowerShell: $env:GEMINI_API_KEY = "..."
node bin/cli.mjs gemini "https://www.youtube.com/watch?v=HoKkasEyDOI"                   # default production-breakdown prompt
node bin/cli.mjs gemini <url|file> --prompt "How is the map animation at 1:20 built?" --start 75 --end 95 --fps 8
node bin/cli.mjs gemini <long-video-url> --agentic                                       # Gemini navigates long videos itself

• Getting a key: create one for free at Google AI Studio. The free tier needs no card but has rate limits (see your limits).

• YouTube links are passed to Gemini directly. They must be public, are in preview at no charge, and the free tier is capped at 8 hours of YouTube video per day. Local files and other sites are uploaded through the Gemini Files API (up to 2 GB per file on the free tier).

• Privacy: on the free tier, Google may use submitted prompts and content to improve its products, including human review; paid-tier usage, and users in the EEA, Switzerland and the UK, are excluded. Don't send private or confidential videos on the free tier. See the Gemini API terms.

• For MCP clients, set the key as an environment variable for the server (for example an "env": { "GEMINI_API_KEY": "..." } block in the MCP config), or set it in your user environment before starting the client. Don't commit keys to repositories.

• The default model is gemini-3.8-flash; override it with --model or VD_GEMINI_MODEL. The result is saved as gemini/gemini-<time>.md in the run folder.

CLI reference

node bin/cli.mjs doctor
node bin/cli.mjs overview <url|file> [--start T] [--end T] [--fps N] [--max-sheets 12]
                          [--max-style-frames 12] [--threshold 0.3] [--soft-threshold 0.35]
                          [--grid 4x4] [--max-height 1080]
                          [--transcribe auto|always|never] [--whisper-model M] [--language L] [--translate]
node bin/cli.mjs zoom     <url|file> --start T --end T [--fps 8] [--grid 4x4] [--tile-width 640] [--max-frames 96]
node bin/cli.mjs frames   <url|file> --timestamps T1,T2,... [--format jpg|png]
node bin/cli.mjs setup-whisper [--model base.en]
node bin/cli.mjs transcribe <url|file> [--start T] [--end T] [--model M] [--language L] [--translate]
node bin/cli.mjs gemini   <url|file> [--prompt TEXT | --prompt-file F] [--model M] [--start T] [--end T] [--fps N] [--agentic]
node bin/cli.mjs mcp

• Times: seconds (75.5), MM:SS(.mmm) or HH:MM:SS. All output timestamps are positions in the original video, even for section downloads.

• --out DIR sets the output folder (default ./vd-output/<video>); --json prints the full result; --captions FILE supplies captions for a local file (otherwise a sidecar name.vtt/name.srt is used).

• Long videos: analyze a section with --start/--end, or keep the default, which caps the overview at 12 storyboard sheets by lowering the sampling rate.

• Downloads are cached per video, so zoom and frames after an overview don't download again.

Environment variables: VD_FFMPEG, VD_FFPROBE, VD_YTDLP (binary paths), VD_CACHE_DIR, VD_OUTPUT_DIR, VD_FONT (TrueType font for timestamps), VD_JS_RUNTIME (yt-dlp --js-runtimes value), VD_WHISPER_MODEL (path to a GGML model), VD_WHISPER_CLI (path to whisper-cli), GEMINI_API_KEY, VD_GEMINI_MODEL, VD_QUIET.

Tested on

Six YouTube videos in different genres, analyzed end to end on Windows 11 ARM64. Full numbers and observations are in docs/TEST-RESULTS.md.

Example output from Spring (© Blender Foundation, CC BY 4.0): one storyboard sheet and the full-film audio strip.

Development

npm test          # node --test: unit tests, synthetic-video pipeline tests, Whisper and Gemini (mock server) tests, MCP protocol test

The pipeline tests generate a synthetic 5-second video with ffmpeg (a hard cut, a wipe, an audio gap and captions) and check every artifact. The Whisper tests synthesize speech with ffmpeg's flite source and run real transcription; they are skipped if no model is installed. The Gemini tests run against a local mock of the API, so they need no key. Tests needing ffmpeg are skipped automatically if it is missing.

• docs/ARCHITECTURE.md: modules, analysis pipeline, ffmpeg recipes, output layout.

• docs/HOW-IT-WAS-BUILT.md: design decisions, problems found during testing and their fixes, limitations, ideas.

• docs/TEST-RESULTS.md: genre validation results.

• docs/examples/kurzgesagt-breakdown.md: a complete three-pass "how it was made" report produced by an agent with these tools.

Troubleshooting

Responsible use

This tool downloads media for personal analysis, study and critique. Respect copyright and the terms of service of the sites you download from. Don't redistribute downloaded videos or frames, and credit creators when you discuss their work.

License

MIT