youtube-mcp

What is this?

An MCP server that lets AI assistants actually watch YouTube videos — not just read transcripts.

It extracts frames at scene changes and visual reference moments, pairs each frame with the exact words spoken at that timestamp, and returns everything as dense interleaved content. The AI sees what's on screen at the exact moment someone says "as you can see here."

No existing YouTube MCP server does this. Every other one is transcript-only. This is the first to combine transcript + vision.

The Token Math

Describe mode gives you 400x savings over raw images — full visual coverage as pure text.

Quick Install

Prerequisites

brew install yt-dlp ffmpeg

Setup

git clone https://github.com/coah80/youtube-mcp.git
cd youtube-mcp
bun install

Add to Claude Code

Add to your ~/.mcp.json:

{
  "mcpServers": {
    "youtube": {
      "command": "bun",
      "args": ["run", "/path/to/youtube-mcp/src/index.ts"]
    }
  }
}

Then restart Claude Code.

Optional: Enable describe mode (local BLIP-2)

This lets the AI "see" every second of a video as text descriptions — no images needed.

cd youtube-mcp
uv venv --python 3.11 .venv
uv pip install --python .venv/bin/python torch torchvision transformers accelerate pillow

First run downloads the BLIP-2 model (~3GB). Subsequent runs use the cached model.

AI Installer

For AI assistants: Copy and paste this into your MCP config to install youtube-mcp.

Add this to ~/.mcp.json (create the file if it doesn't exist):

{
  "mcpServers": {
    "youtube": {
      "command": "bun",
      "args": ["run", "/path/to/youtube-mcp/src/index.ts"]
    }
  }
}

Replace /path/to/youtube-mcp with the actual clone path.

If the file already has other servers, just add the "youtube" key inside the existing "mcpServers" object.

How It Works

YouTube URL
    │
    ├──→ yt-dlp ──→ Transcript (timestamped, word-level)
    │
    ├──→ yt-dlp ──→ Stream URL ──→ ffmpeg ──→ Frames
    │                                │
    │                    ┌────────────┼────────────┐
    │                    │            │            │
    │              Scene Change  Visual Cues  Regular
    │              Detection    in Transcript  Intervals
    │              (ffmpeg)     ("as you can   (fill gaps)
    │                           see here")
    │                    │            │            │
    │                    └────────────┼────────────┘
    │                                │
    │                    Frame Selection (prioritized)
    │                                │
    └──────────────────→ Dense Interleave
                              │
                   ┌──────────┴──────────┐
                   │                     │
              Image Mode            Describe Mode
           (raw screenshots)     (BLIP-2 captions)
                   │                     │
              Frame + "words         Text description
              spoken during          + "words spoken
              this frame"            during this frame"

Visual Cue Detection

The analyzer scans transcript text for 25+ patterns indicating the speaker is referencing something visual:

When detected, a frame is extracted at that exact timestamp — so the AI sees what the speaker was pointing at.

Scene Change Detection

Uses ffmpeg's scene detection filter (select=gt(scene,0.3)) to find where the visual content actually changes. This means:

• Static talking-head sections get fewer frames (nothing's changing)

• Slide transitions, screen recordings, demos get more frames (lots changing)

Segment-Based Processing

For videos longer than 5 minutes, watch_video processes in 3-minute segments with ~1 frame every 5 seconds. The AI calls it repeatedly:

watch_video(url) → first 3 min, 36 frames
watch_video(url, start_time=180) → next 3 min, 36 frames
watch_video(url, start_time=360) → next 3 min, 36 frames
...until the end

Each response tells the AI how to continue: "To continue watching, call watch_video with start_time=360"

Tools

Examples

"Watch this video and summarize it"

The AI calls watch_video and gets interleaved content like:

[1:23] (scene change) "and here's where it gets interesting"
[screenshot of code editor]

[1:28] "if you look at this function right here"
[screenshot showing the function being discussed]

[1:33] (visual reference) "notice how the state updates"
[screenshot at the exact moment they reference the visual]

"Describe this entire lecture for me"

The AI calls describe_video and gets pure text:

[0:00] [VISUAL] A title slide reading "Introduction to Neural Networks"
[0:00] Welcome everyone to today's lecture on neural networks.
[0:05] [VISUAL] A diagram showing interconnected nodes in layers
[0:05] We'll start with the basic architecture.
[0:10] [VISUAL] The same diagram with arrows highlighted between layers
[0:10] Each connection between nodes has a weight...

600 frames of a 10-minute video → ~18K tokens. Fits in any context window.

Architecture

youtube-mcp/
├── src/
│   ├── index.ts        # MCP server — 6 tool definitions
│   ├── youtube.ts      # yt-dlp + ffmpeg operations (stream URL, frames, scenes)
│   ├── analyzer.ts     # Visual cue detection, chunking, dense interleaving
│   ├── describe.ts     # BLIP-2 integration (TypeScript wrapper)
│   └── captioner.py    # BLIP-2 inference (Python, runs on MPS/CUDA/CPU)
├── .venv/              # Python venv for BLIP-2 (optional)
├── package.json
├── tsconfig.json
└── README.md

Tech Stack

• Runtime: Bun

• MCP SDK: @modelcontextprotocol/sdk

• Video: yt-dlp + ffmpeg

• Vision (optional): BLIP-2 via PyTorch on Apple MPS

Compatibility

Works with any MCP-compatible AI assistant:

• Claude Code (CLI, Desktop, Web)

• Claude Desktop

• Cursor

• Any future MCP host

The image-based tools (watch_video, get_frames, get_scene_overview) require a vision-capable model.

The text-based tool (describe_video) works with any model — even text-only ones — because BLIP-2 converts all visuals to text locally.

Roadmap

• Gemini Flash proxy mode — use Gemini Flash ($0.10/1M tokens) as a visual encoder for higher-quality frame descriptions than BLIP-2

• Frame deduplication — perceptual similarity hashing to skip near-identical frames

• Keyframe extraction — use ffmpeg I-frame detection instead of fixed intervals

• Whisper integration — local audio transcription when YouTube captions aren't available

• Timestamp burning — burn MM:SS into frame pixels (requires ffmpeg with libfreetype)

• npm package — npx youtube-mcp one-liner install

Research

This project was informed by deep research into how Gemini, GPT-4o, and open-source tools handle video:

• Gemini processes video at 1 FPS using SigLIP-SO400M (258 tokens/frame) with native multimodal attention

• GPT-4o sends base64 JPEG frames via the vision API (~12K tokens/frame)

• No existing YouTube MCP server combines transcript + frame extraction — this is the first

Key references: LiveCC (CVPR 2025), mcp-deep-video, videostil, llm-video-frames

License

MIT