# Vision MCP Server

MCP server for image/video understanding and audio transcription. Uses GLM-4.6V-Flash for visual analysis and faster-whisper for audio transcription.

> This repository is a personal/demo project. It integrates external models and media-download tools; check each dependency and model license before commercial use. Keep `.env` local and never commit API keys or runtime logs.

## Screenshots

### Web workflow

![Video summary workflow](docs/screenshots/vision-home.png)

### Summary result

![Structured video summary result](docs/screenshots/vision-result.png)

### Frame analysis

![Frame-by-frame analysis and overall summary](docs/screenshots/vision-detail.png)

The screenshots show a real end-to-end Bilibili video test using the Web interface. The summarized topic belongs to the test video and is not bundled project content.

## Why this project

Long videos often contain useful information but are expensive to watch from beginning to end. This project turns a video into a compact, searchable understanding result by combining visual frames, speech, subtitles, and a final language-model summary.

## Architecture

```text
Web UI
  -> Hono HTTP/SSE layer
  -> media acquisition and type detection
  -> FFmpeg frame/audio preprocessing
  -> GLM vision analysis + faster-whisper transcription
  -> structured summary response
```

The MCP layer exposes the same capabilities to AI clients, while the Web UI provides a human-facing workflow with progress events. Visual analysis and transcription can run in parallel when the input supports both.

## Typical workflow

1. Paste a public video URL or provide a local media file.
2. Choose `smart`, `quick`, or `visual` mode and optionally describe the information to focus on.
3. The server detects the media type, downloads or reads it, extracts representative frames, and prepares audio.
4. GLM analyzes visual content and faster-whisper produces a timestamped transcript.
5. The result is combined into a structured summary and displayed through the Web UI or returned by MCP.

## Validation

- TypeScript build completed with `npm run build`.
- End-to-end validation completed with a public Bilibili video.
- Verified progress events for frame analysis, transcription, summary generation, and completion.
- Tested timeout, VAD, dynamic-module, and failed-audio paths during iteration.

## Scope and limitations

- This is a personal/demo project, not a hosted public inference service.
- API credentials, model weights, runtime logs, and generated media are intentionally excluded from Git.
- Long videos and CPU transcription can be slow and require larger timeouts.
- Video downloading must comply with the source platform's terms and applicable copyright rules.

## Tools

| Tool | Description |
|------|-------------|
| `describe_image` | Describe an image (URL, base64, or local file) |
| `describe_video` | Describe a video with visual analysis + optional audio transcription |
| `describe_file` | Auto-detect file type (image/video/audio) and describe |
| `transcribe_audio` | Transcribe audio from a video or audio file using Whisper |

## Prerequisites

- **Node.js** 18+
- **ffmpeg** (for video frame extraction and audio processing)
- **ffprobe** (for video duration detection, usually bundled with ffmpeg)
- **Python 3.8+** with `faster-whisper` installed (`pip install faster-whisper`)
- **Zhipu API key** (for GLM-4.6V-Flash vision model)

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `ZHIPU_API_KEY` | *(required)* | Zhipu API key for GLM-4.6V-Flash |
| `PYTHON_PATH` | `python` | Python interpreter path (e.g. `D:\miniconda\python.exe`) |
| `FFMPEG_PATH` | `ffmpeg` | ffmpeg executable path |
| `FFPROBE_PATH` | `ffprobe` | ffprobe executable path |
| `YTDLP_PATH` | `yt-dlp` | yt-dlp executable path (for non-Bilibili video downloads) |
| `WHISPER_MODEL` | `medium` | Whisper model size (`tiny`, `base`, `small`, `medium`, `large-v3`) |
| `WHISPER_TIMEOUT` | `300000` | Whisper transcription timeout in ms |
| `VIDEO_DOWNLOAD_TIMEOUT` | `60000` | Video download timeout in ms |
| `API_TIMEOUT` | `90000` | Single GLM API call timeout in ms |

## Register MCP Server

### Claude Code

Edit `~/.claude/.mcp.json`:

```json
{
  "mcpServers": {
    "vision": {
      "command": "node",
      "args": ["/path/to/vision-mcp-server/dist/index.js"],
      "env": {
        "ZHIPU_API_KEY": "your-api-key",
        "PYTHON_PATH": "python",
        "FFMPEG_PATH": "ffmpeg",
        "FFPROBE_PATH": "ffprobe"
      }
    }
  }
}
```

### Codex CLI

```bash
codex mcp add vision --env ZHIPU_API_KEY=your-api-key --env PYTHON_PATH=python --env FFMPEG_PATH=ffmpeg -- node /path/to/vision-mcp-server/dist/index.js
```

Or edit `~/.codex/config.toml`:

```toml
[mcp_servers.vision]
command = "node"
args = ["/path/to/vision-mcp-server/dist/index.js"]

[mcp_servers.vision.env]
ZHIPU_API_KEY = "your-api-key"
PYTHON_PATH = "python"
FFMPEG_PATH = "ffmpeg"
FFPROBE_PATH = "ffprobe"
```

### OpenCode

Edit `~/.config/opencode/opencode.json` or your project's `opencode.json`:

```json
{
  "mcp": {
    "vision": {
      "type": "local",
      "enabled": true,
      "command": ["node", "/path/to/vision-mcp-server/dist/index.js"],
      "environment": {
        "ZHIPU_API_KEY": "your-api-key",
        "PYTHON_PATH": "python",
        "FFMPEG_PATH": "ffmpeg",
        "FFPROBE_PATH": "ffprobe"
      },
      "timeout": 300000
    }
  }
}
```

> **Note:** OpenCode's `timeout` field controls how long it waits for MCP tool responses. Set to at least 300000 (5 minutes) for video processing with Whisper transcription.

## Windows Example

On Windows, if ffmpeg/python aren't on PATH, specify full paths:

```json
{
  "env": {
    "ZHIPU_API_KEY": "your-api-key",
    "PYTHON_PATH": "D:\\miniconda\\python.exe",
    "FFMPEG_PATH": "C:\\Users\\You\\AppData\\Local\\Microsoft\\WinGet\\Packages\\Gyan.FFmpeg_...\\bin\\ffmpeg.exe",
    "FFPROBE_PATH": "C:\\Users\\You\\AppData\\Local\\Microsoft\\WinGet\\Packages\\Gyan.FFmpeg_...\\bin\\ffprobe.exe"
  }
}
```

## Build

```bash
npm install
npx tsc
```

## Public repository notes

- Copy `.env.example` to `.env` and fill in local values before running.
- The `.env`, `web.log`, `dist/`, and `node_modules/` paths are intentionally excluded from Git.
- This project does not train or distribute a vision model. It orchestrates external vision APIs, video processing, and speech recognition.
- Long-video transcription can be slow on CPU and may require a longer MCP or Web request timeout.

## How It Works

### describe_video (online video)

1. **Bilibili**: Download via API → extract base64 → call GLM-4V video API + Whisper in parallel
2. **Other URLs**: Try GLM native video API first → fallback to yt-dlp download → frame extraction + Whisper
3. **Local file**: Extract key frames (adaptive interval) → batch GLM-4V analysis + Whisper in parallel

### Frame extraction strategy

- Dynamic interval based on video duration (3-30 seconds per frame)
- Target ~12 frames, max 15 frames
- Short videos (< 1 min): 3s/frame for detail
- Long videos (5+ min): 15-30s/frame for coverage

### transcribe_audio

1. Extract audio to WAV with ffmpeg (16kHz mono)
2. Run faster-whisper via Python subprocess
3. Return timestamped transcript