Skip to main content
Glama

video-storyboard-mcp

中文文档

Portable MCP server, CLI, and Antigravity plugin built on vendored SAOS-3624/dsh-video-storyboard (v0.7.6, Apache-2.0). Watch long local or online videos (Bilibili, YouTube, etc.) in Antigravity, Claude Code, Cursor, Gemini CLI, or Codex using a single compact storyboard image + timestamped transcript, with zero third-party npm dependencies.

Features

  • Variable-Tier Scene Storyboards (1/1..1/8): One representative frame per shot, scaled by cut-point density (1/1 for long takes, 1/8 for rapid cuts) and DP row-packed onto tightly cropped canvases (~222 vision tokens per default sheet).

  • Audio + Multi-Source Transcript Alignment: Renders a bottom loudness waveform strip (audio="present") and resolves transcripts cleanly from site CC subtitles, embedded subtitle tracks, local ASR, or targeted burned-in RapidOCR.

  • Built-In Offline sherpa-onnx ASR Lifecycle: Includes a zero-framework stdlib HTTP ASR server (server/asr_server.py) and one-command installer (setup-asr) supporting SenseVoice (zh/en/ja/ko/yue) and Paraformer + Silero VAD with automatic launch and 5-second idle shutdown.

  • Native Clip & Cache Hygiene Extensions: Adds video_clip (export short .mp4/.mp3 slices for native multimodal inspection) and video_cache_clean (wipe downloaded video caches after watching).

  • Bilibili Proxy Split & Error Surfacing: Automatically routes domestic Bilibili hosts (bilibili.com, b23.tv, bilivideo.com, hdslb.com) around HTTP_PROXY/HTTPS_PROXY while preserving proxies for international CDNs, with --proxy "" fallback.

Related MCP server: watch-mcp

8 Tools Exposed Over MCP & CLI

Tool

Purpose

Image Tokens

video_probe

Inspect duration, resolution, FPS, tracks, and available link heights (no download)

0

video_storyboard

Generate variable-tier DP-packed storyboard sheet(s) + aligned transcript + dryRun estimator

~222 / sheet

video_zoom

Re-read a cellId or [from, to] window at higher resolution

~222

video_locate

Fuzzy search spoken transcript lines to find exact timestamps

0

video_subtitles

Crop the burned-in subtitle band vertically across timestamps into one strip image

~222

video_help

Print the upstream parameter, strategy, tierMap, and cost manual

0

video_clip

Export trimmed .mp4 or .mp3 slice (max 90s) for native inspection

0

video_cache_clean

Delete cached downloaded videos, clips, and preview images

0

Prerequisites

  • Node.js >= 20

  • ffmpeg + ffprobe (winget install Gyan.FFmpeg / brew install ffmpeg / apt install ffmpeg)

  • yt-dlp (optional, for URLs such as Bilibili / YouTube: pip install --user -U yt-dlp)

  • Python >= 3.9 (optional, for setup-asr offline speech transcription and optional rapidocr hard-subtitle checking)

3-Step Quick Start

1. Install into Antigravity or print your MCP config

# Antigravity global plugin (~/.gemini/config/plugins/video-storyboard/)
node scripts/install.mjs --antigravity-global

# Or install into a specific workspace (.agents/plugins/video-storyboard/)
node scripts/install.mjs --antigravity-workspace /path/to/workspace

# Or print ready-to-paste MCP config for Claude Code / Cursor / Gemini CLI / Codex
node scripts/install.mjs --print-mcp

2. Run diagnostics

node server/cli.mjs doctor

3. (Optional) Enable offline speech transcription (sherpa-onnx)

By default, videos without subtitle tracks report visual storyboards cleanly without transcribing audio (asr-status shows not configured). To add free offline transcription (downloads ~156 MB SenseVoice int8 + Silero VAD into .asr-runtime/):

# Preview exact URLs and paths without downloading
node server/cli.mjs setup-asr --dry-run

# Install local sherpa-onnx + SenseVoice int8 + Silero VAD and configure auto-lifecycle
node server/cli.mjs setup-asr

# Check active ASR state
node server/cli.mjs asr-status

Uses DSH_ASR_FALLBACK_CONFIG when neither ~/.dsh/asr.json nor $DSH_HOME/asr.json exists, so any existing user ASR config takes precedence automatically.

CLI Examples

node server/cli.mjs probe "https://b23.tv/..." --allow-anonymous
node server/cli.mjs storyboard "/path/to/video.mp4" --strategy auto
node server/cli.mjs zoom "/path/to/video.mp4" --cell 2
node server/cli.mjs locate "/path/to/video.mp4" "keyword"
node server/cli.mjs subtitles "/path/to/video.mp4" --max-strips 12
node server/cli.mjs clip "/path/to/video.mp4" --cell 2 --audio-only
node server/cli.mjs clean --all --include-scan-cache

Notes on Auth Cookies & Cost

  • Anonymous Bilibili/YouTube links may be capped or rate-limited (HTTP 403 / 567). For higher bitrates or platform CC tracks, export a Netscape cookies.txt and configure {"cookies": "/path/to/cookies.txt"} in ~/.dsh/video.json (or pass --cookies). Never commit cookies.txt to version control.

  • Use video_probe first and dryRun: true on long videos before committing sheets. Use from + segment: 300 to scan long videos in 5-minute windows.

Running Tests

npm test

Includes offline smoke coverage for MCP stdio JSON-RPC (8 tools), setup-asr --dry-run, install.mjs --print-mcp, and end-to-end asr_server.py --mock auto-launch + transcription + idle shutdown.

Attribution & License

Licensed under the Apache License, Version 2.0. Built on vendored source from SAOS-3624/dsh-video-storyboard (v0.7.6). See LICENSE, NOTICE, and vendor/dsh-video-storyboard/CREDITS.md for upstream copyright, third-party credits, and documented modifications.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      A
      quality
      B
      maintenance
      Enables Claude Code, Codex, Claude Desktop and Cursor to see and hear MP4/MOV/MKV/WebM videos and screen recordings through timestamped keyframe storyboards, zoom and full-resolution frame drill-down, and a locally generated Whisper transcript. It also works with recording bundles that capture clicks and typed context, and runs entirely on the user's machine with no uploads or API keys.
      5
      1
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Enables local video understanding by generating numbered storyboard sheets with timecoded transcripts, zooming into selected segments, locating spoken phrases, probing media metadata, and tracking playback progress. It uses local ffmpeg and optional offline speech recognition.
      MIT