Vision MCP Server

MCP server for image/video understanding and audio transcription. Uses GLM-4.6V-Flash for visual analysis and faster-whisper for audio transcription.

This repository is a personal/demo project. It integrates external models and media-download tools; check each dependency and model license before commercial use. Keep .env local and never commit API keys or runtime logs.

Screenshots

Web workflow

Summary result

Frame analysis

The screenshots show a real end-to-end Bilibili video test using the Web interface. The summarized topic belongs to the test video and is not bundled project content.

Related MCP server: qwen-omni-mcp

Why this project

Long videos often contain useful information but are expensive to watch from beginning to end. This project turns a video into a compact, searchable understanding result by combining visual frames, speech, subtitles, and a final language-model summary.

Architecture

Web UI
  -> Hono HTTP/SSE layer
  -> media acquisition and type detection
  -> FFmpeg frame/audio preprocessing
  -> GLM vision analysis + faster-whisper transcription
  -> structured summary response

The MCP layer exposes the same capabilities to AI clients, while the Web UI provides a human-facing workflow with progress events. Visual analysis and transcription can run in parallel when the input supports both.

Typical workflow

1. Paste a public video URL or provide a local media file.

2. Choose smart, quick, or visual mode and optionally describe the information to focus on.

3. The server detects the media type, downloads or reads it, extracts representative frames, and prepares audio.

4. GLM analyzes visual content and faster-whisper produces a timestamped transcript.

5. The result is combined into a structured summary and displayed through the Web UI or returned by MCP.

Validation

• TypeScript build completed with npm run build.

• End-to-end validation completed with a public Bilibili video.

• Verified progress events for frame analysis, transcription, summary generation, and completion.

• Tested timeout, VAD, dynamic-module, and failed-audio paths during iteration.

Scope and limitations

• This is a personal/demo project, not a hosted public inference service.

• API credentials, model weights, runtime logs, and generated media are intentionally excluded from Git.

• Long videos and CPU transcription can be slow and require larger timeouts.

• Video downloading must comply with the source platform's terms and applicable copyright rules.

Tools

Prerequisites

• Node.js 18+

• ffmpeg (for video frame extraction and audio processing)

• ffprobe (for video duration detection, usually bundled with ffmpeg)

• Python 3.8+ with faster-whisper installed (pip install faster-whisper)

• Zhipu API key (for GLM-4.6V-Flash vision model)

Environment Variables

Register MCP Server

Claude Code

Edit ~/.claude/.mcp.json:

{
  "mcpServers": {
    "vision": {
      "command": "node",
      "args": ["/path/to/vision-mcp-server/dist/index.js"],
      "env": {
        "ZHIPU_API_KEY": "your-api-key",
        "PYTHON_PATH": "python",
        "FFMPEG_PATH": "ffmpeg",
        "FFPROBE_PATH": "ffprobe"
      }
    }
  }
}

Codex CLI

codex mcp add vision --env ZHIPU_API_KEY=your-api-key --env PYTHON_PATH=python --env FFMPEG_PATH=ffmpeg -- node /path/to/vision-mcp-server/dist/index.js

Or edit ~/.codex/config.toml:

[mcp_servers.vision]
command = "node"
args = ["/path/to/vision-mcp-server/dist/index.js"]

[mcp_servers.vision.env]
ZHIPU_API_KEY = "your-api-key"
PYTHON_PATH = "python"
FFMPEG_PATH = "ffmpeg"
FFPROBE_PATH = "ffprobe"

OpenCode

Edit ~/.config/opencode/opencode.json or your project's opencode.json:

{
  "mcp": {
    "vision": {
      "type": "local",
      "enabled": true,
      "command": ["node", "/path/to/vision-mcp-server/dist/index.js"],
      "environment": {
        "ZHIPU_API_KEY": "your-api-key",
        "PYTHON_PATH": "python",
        "FFMPEG_PATH": "ffmpeg",
        "FFPROBE_PATH": "ffprobe"
      },
      "timeout": 300000
    }
  }
}

Note: OpenCode's timeout field controls how long it waits for MCP tool responses. Set to at least 300000 (5 minutes) for video processing with Whisper transcription.

Windows Example

On Windows, if ffmpeg/python aren't on PATH, specify full paths:

{
  "env": {
    "ZHIPU_API_KEY": "your-api-key",
    "PYTHON_PATH": "D:\\miniconda\\python.exe",
    "FFMPEG_PATH": "C:\\Users\\You\\AppData\\Local\\Microsoft\\WinGet\\Packages\\Gyan.FFmpeg_...\\bin\\ffmpeg.exe",
    "FFPROBE_PATH": "C:\\Users\\You\\AppData\\Local\\Microsoft\\WinGet\\Packages\\Gyan.FFmpeg_...\\bin\\ffprobe.exe"
  }
}

Build

npm install
npx tsc

Public repository notes

• Copy .env.example to .env and fill in local values before running.

• The .env, web.log, dist/, and node_modules/ paths are intentionally excluded from Git.

• This project does not train or distribute a vision model. It orchestrates external vision APIs, video processing, and speech recognition.

• Long-video transcription can be slow on CPU and may require a longer MCP or Web request timeout.

How It Works

describe_video (online video)

1. Bilibili: Download via API → extract base64 → call GLM-4V video API + Whisper in parallel

2. Other URLs: Try GLM native video API first → fallback to yt-dlp download → frame extraction + Whisper

3. Local file: Extract key frames (adaptive interval) → batch GLM-4V analysis + Whisper in parallel

Frame extraction strategy

• Dynamic interval based on video duration (3-30 seconds per frame)

• Target ~12 frames, max 15 frames

• Short videos (< 1 min): 3s/frame for detail

• Long videos (5+ min): 15-30s/frame for coverage

transcribe_audio

1. Extract audio to WAV with ffmpeg (16kHz mono)

2. Run faster-whisper via Python subprocess

3. Return timestamped transcript