VibeVoice-ASR Server

Local speech-to-text using Microsoft's VibeVoice-ASR model. Run it as an OpenAI-compatible API server or as an MCP server that plugs directly into Claude Code, OpenCode, Cursor, and other AI tools.

• Automatic speaker diarization

• Timestamps on every segment

• Output as plain text, JSON, SRT, or VTT

• Runs on CUDA, Apple Silicon (MPS), or CPU

• Model downloads automatically on first run

Requirements

• Python 3.10+

• FFmpeg (used by the model's audio processor)

Install FFmpeg:

# macOS
brew install ffmpeg

# Ubuntu / Debian
sudo apt-get install ffmpeg

# Windows (with Chocolatey)
choco install ffmpeg

Related MCP server: Claude Voice Commands

Quick Start

# Clone the repo
git clone https://github.com/tjameswilliams/vibevoice-server.git
cd vibevoice-server

# Create a virtual environment (recommended)
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate

# Install
pip install -e .

# For NVIDIA GPU acceleration (optional)
pip install -e ".[cuda]"

The first time you run either the API server or MCP server, the model (~3 GB) will be downloaded from HuggingFace and cached locally.

Option 1: OpenAI-Compatible API Server

Start the server:

vibevoice-server

The server starts on http://localhost:8000 by default. It exposes the same endpoint shape as the OpenAI Audio API, so any client library or tool that speaks that protocol works out of the box.

CLI Options

vibevoice-server [OPTIONS]

  --host        Bind address (default: 0.0.0.0)
  --port        Bind port (default: 8000)
  --device      Device: auto, cuda, mps, cpu (default: auto)
  --dtype       Data type: auto, bfloat16, float32 (default: auto)
  --log-level   Log level: debug, info, warning, error (default: info)

Transcribe Audio

curl http://localhost:8000/v1/audio/transcriptions \
  -F file=@meeting.wav \
  -F response_format=verbose_json

Parameters:

Response Formats

json (default):

{"text": "Hello, welcome to the meeting."}

verbose_json — includes timestamps, speaker IDs, and segments:

{
  "task": "transcribe",
  "language": "en",
  "duration": 12.5,
  "text": "Hello, welcome to the meeting.",
  "segments": [
    {"id": 0, "start": 0.0, "end": 3.2, "text": "Hello, welcome to the meeting.", "speaker": 0}
  ]
}

srt and vtt — subtitle formats with speaker labels, ready to use with video players.

text — plain transcript string, no JSON wrapper.

Other Endpoints

# List models
curl http://localhost:8000/v1/models

# Health check
curl http://localhost:8000/health

Using with OpenAI Client Libraries

Point any OpenAI SDK at your local server:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

with open("recording.wav", "rb") as f:
    transcript = client.audio.transcriptions.create(
        model="vibevoice-asr",
        file=f,
        response_format="verbose_json",
    )

print(transcript.text)

Docker

# Build
docker build -t vibevoice-server .

# Run (CPU)
docker run -p 8000:8000 -v vibevoice-cache:/models vibevoice-server

# Run (NVIDIA GPU)
docker run --gpus all -p 8000:8000 -v vibevoice-cache:/models vibevoice-server

Option 2: MCP Server

The MCP (Model Context Protocol) server lets AI tools call transcription directly — no HTTP server needed. The model runs in the same process as the MCP server.

MCP Tools

transcribe_audio parameters:

Claude Code

Add to your project's .mcp.json (or ~/.claude/mcp.json for global access):

{
  "mcpServers": {
    "vibevoice-asr": {
      "command": "vibevoice-mcp",
      "args": []
    }
  }
}

With device override:

{
  "mcpServers": {
    "vibevoice-asr": {
      "command": "vibevoice-mcp",
      "args": ["--device", "mps"]
    }
  }
}

Restart Claude Code after adding the config. The three tools (transcribe_audio, load_vibevoice_model, get_vibevoice_status) will appear automatically.

Cursor

Add to .cursor/mcp.json in your project root:

{
  "mcpServers": {
    "vibevoice-asr": {
      "command": "vibevoice-mcp",
      "args": []
    }
  }
}

OpenCode

Add to your OpenCode MCP configuration (opencode.json or via settings):

{
  "mcpServers": {
    "vibevoice-asr": {
      "command": "vibevoice-mcp",
      "args": []
    }
  }
}

Any MCP-Compatible Tool

The server uses stdio transport — the standard for local MCP servers. Any tool that supports MCP can run it with:

• Command: vibevoice-mcp

• Args: [] (optional: ["--device", "mps"] or ["--device", "cuda"])

• Transport: stdio

The MCP server reads JSON-RPC from stdin and writes responses to stdout. All logs go to stderr.

MCP CLI Options

vibevoice-mcp [OPTIONS]

  --device      Device: auto, cuda, mps, cpu (default: auto)
  --dtype       Data type: auto, bfloat16, float32 (default: auto)
  --log-level   Log level (default: warning)

Configuration

All settings can be controlled via environment variables (prefixed with VIBEVOICE_), CLI flags, or a .env file. See .env.example for the full list.

Device auto-detection picks the best available: CUDA > MPS > CPU.

Hardware Notes

The model is ~3 GB. First load takes 60-90 seconds (downloading + loading weights). Subsequent starts are faster when cached.

License

MIT