Skip to main content
Glama

VibeVoice-ASR Server

Local speech-to-text using Microsoft's VibeVoice-ASR model. Run it as an OpenAI-compatible API server or as an MCP server that plugs directly into Claude Code, OpenCode, Cursor, and other AI tools.

  • Automatic speaker diarization

  • Timestamps on every segment

  • Output as plain text, JSON, SRT, or VTT

  • Runs on CUDA, Apple Silicon (MPS), or CPU

  • Model downloads automatically on first run

Requirements

  • Python 3.10+

  • FFmpeg (used by the model's audio processor)

Install FFmpeg:

# macOS
brew install ffmpeg

# Ubuntu / Debian
sudo apt-get install ffmpeg

# Windows (with Chocolatey)
choco install ffmpeg

Related MCP server: livechat-mcp

Quick Start

# Clone the repo
git clone https://github.com/tjameswilliams/vibevoice-server.git
cd vibevoice-server

# Create a virtual environment (recommended)
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate

# Install
pip install -e .

# For NVIDIA GPU acceleration (optional)
pip install -e ".[cuda]"

The first time you run either the API server or MCP server, the model (~3 GB) will be downloaded from HuggingFace and cached locally.


Option 1: OpenAI-Compatible API Server

Start the server:

vibevoice-server

The server starts on http://localhost:8000 by default. It exposes the same endpoint shape as the OpenAI Audio API, so any client library or tool that speaks that protocol works out of the box.

CLI Options

vibevoice-server [OPTIONS]

  --host        Bind address (default: 0.0.0.0)
  --port        Bind port (default: 8000)
  --device      Device: auto, cuda, mps, cpu (default: auto)
  --dtype       Data type: auto, bfloat16, float32 (default: auto)
  --log-level   Log level: debug, info, warning, error (default: info)

Transcribe Audio

curl http://localhost:8000/v1/audio/transcriptions \
  -F file=@meeting.wav \
  -F response_format=verbose_json

Parameters:

Parameter

Type

Default

Description

file

file

required

Audio file (wav, mp3, flac, m4a, ogg, etc.)

model

string

vibevoice-asr

Model identifier (accepted but ignored)

response_format

string

json

text, json, verbose_json, srt, vtt

prompt

string

Optional context to guide transcription

language

string

Language code (used in verbose_json output)

Response Formats

json (default):

{"text": "Hello, welcome to the meeting."}

verbose_json — includes timestamps, speaker IDs, and segments:

{
  "task": "transcribe",
  "language": "en",
  "duration": 12.5,
  "text": "Hello, welcome to the meeting.",
  "segments": [
    {"id": 0, "start": 0.0, "end": 3.2, "text": "Hello, welcome to the meeting.", "speaker": 0}
  ]
}

srt and vtt — subtitle formats with speaker labels, ready to use with video players.

text — plain transcript string, no JSON wrapper.

Other Endpoints

# List models
curl http://localhost:8000/v1/models

# Health check
curl http://localhost:8000/health

Using with OpenAI Client Libraries

Point any OpenAI SDK at your local server:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

with open("recording.wav", "rb") as f:
    transcript = client.audio.transcriptions.create(
        model="vibevoice-asr",
        file=f,
        response_format="verbose_json",
    )

print(transcript.text)

Docker

# Build
docker build -t vibevoice-server .

# Run (CPU)
docker run -p 8000:8000 -v vibevoice-cache:/models vibevoice-server

# Run (NVIDIA GPU)
docker run --gpus all -p 8000:8000 -v vibevoice-cache:/models vibevoice-server

Option 2: MCP Server

The MCP (Model Context Protocol) server lets AI tools call transcription directly — no HTTP server needed. The model runs in the same process as the MCP server.

MCP Tools

Tool

Description

transcribe_audio

Transcribe an audio file. Pass an absolute file path and get back the transcript.

load_vibevoice_model

Pre-load the model into memory (~60-90s). Optional — the model loads automatically on first transcription.

get_vibevoice_status

Check whether the model is loaded, and which device/dtype it's using.

transcribe_audio parameters:

Parameter

Type

Default

Description

file_path

string

required

Absolute path to the audio file

response_format

string

text

text, json, verbose_json, srt, vtt

prompt

string

Optional context to guide transcription

language

string

Language code (for verbose_json output)

Claude Code

Add to your project's .mcp.json (or ~/.claude/mcp.json for global access):

{
  "mcpServers": {
    "vibevoice-asr": {
      "command": "vibevoice-mcp",
      "args": []
    }
  }
}

With device override:

{
  "mcpServers": {
    "vibevoice-asr": {
      "command": "vibevoice-mcp",
      "args": ["--device", "mps"]
    }
  }
}

Restart Claude Code after adding the config. The three tools (transcribe_audio, load_vibevoice_model, get_vibevoice_status) will appear automatically.

Cursor

Add to .cursor/mcp.json in your project root:

{
  "mcpServers": {
    "vibevoice-asr": {
      "command": "vibevoice-mcp",
      "args": []
    }
  }
}

OpenCode

Add to your OpenCode MCP configuration (opencode.json or via settings):

{
  "mcpServers": {
    "vibevoice-asr": {
      "command": "vibevoice-mcp",
      "args": []
    }
  }
}

Any MCP-Compatible Tool

The server uses stdio transport — the standard for local MCP servers. Any tool that supports MCP can run it with:

  • Command: vibevoice-mcp

  • Args: [] (optional: ["--device", "mps"] or ["--device", "cuda"])

  • Transport: stdio

The MCP server reads JSON-RPC from stdin and writes responses to stdout. All logs go to stderr.

MCP CLI Options

vibevoice-mcp [OPTIONS]

  --device      Device: auto, cuda, mps, cpu (default: auto)
  --dtype       Data type: auto, bfloat16, float32 (default: auto)
  --log-level   Log level (default: warning)

Configuration

All settings can be controlled via environment variables (prefixed with VIBEVOICE_), CLI flags, or a .env file. See .env.example for the full list.

Variable

Default

Description

VIBEVOICE_DEVICE

auto

auto, cuda, mps, cpu

VIBEVOICE_DTYPE

auto

auto, bfloat16, float32

VIBEVOICE_CACHE_DIR

(HuggingFace default)

Where to store downloaded model weights

VIBEVOICE_MODEL_ID

microsoft/VibeVoice-ASR-HF

HuggingFace model ID

VIBEVOICE_HOST

0.0.0.0

API server bind address

VIBEVOICE_PORT

8000

API server bind port

VIBEVOICE_LOG_LEVEL

info

Logging level

Device auto-detection picks the best available: CUDA > MPS > CPU.


Hardware Notes

Platform

Device

Dtype

Notes

NVIDIA GPU

cuda

bfloat16

Fastest. Flash Attention 2 enabled automatically. Install with .[cuda].

Apple Silicon

mps

float32

Works well on M1/M2/M3/M4.

CPU

cpu

float32

Slower but works everywhere.

The model is ~3 GB. First load takes 60-90 seconds (downloading + loading weights). Subsequent starts are faster when cached.


License

MIT

Available Tools

3 tools
get_vibevoice_statusA

Check the current status of the VibeVoice-ASR server.

Returns model loaded state, device, dtype, and version info.

Inputs: no named parameters. Returns: result

load_vibevoice_modelA

Pre-load the VibeVoice-ASR model into memory.

Call this before transcribing to avoid a long wait on the first transcription. Model loading takes ~60-90 seconds depending on hardware.

Inputs: no named parameters. Returns: result

transcribe_audioA

Transcribe an audio file using VibeVoice-ASR.

Args: file_path: Absolute path to the audio file (wav, mp3, flac, etc.) response_format: Output format: text, json, verbose_json, srt, vtt (default: text) prompt: Optional prompt/context to guide transcription language: Language code for verbose_json output (default: en)

Inputs: file_path (required), prompt, language, response_format. Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A4.1/5.0

    Scored across 3 tools

    Disambiguation5/5

    Each tool has a unique purpose: checking server status, loading the model, and transcribing audio. No functional overlap, making it clear which tool to use for each task.

    Naming Consistency5/5

    All tool names follow a consistent verb_noun pattern in snake_case (get_vibevoice_status, load_vibevoice_model, transcribe_audio), making them predictable and easy to remember.

    Tool Count5/5

    Three tools cover the essential operations of an ASR server—status check, model loading, and transcription. The count is well-scoped for the domain without unnecessary extras.

    Completeness4/5

    The set covers the core workflow (status, load, transcribe). A minor gap is the lack of an unload model tool, but this does not severely hinder functionality for typical use cases.

    Maintenance

    ActivityInactive
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers

    • A
      license
      Not graded
      quality
      D
      maintenance
      Enables bidirectional voice interaction for Claude Code using local speech-to-text and text-to-speech models optimized for Apple Silicon. It provides tools to listen to user speech via microphone and speak responses aloud through system speakers.
      16
      Apache 2.0
    • A
      license
      A
      quality
      D
      maintenance
      Enables continuous voice conversation with AI coding assistants by locally transcribing speech with Whisper and delivering utterances as text prompts.
      4
      1
      MIT
    • A
      license
      Not graded
      quality
      C
      maintenance
      Local-first meeting capture and transcription for Claude Code. Records audio from meeting apps, transcribes locally with whisper.cpp, and produces structured notes via Claude.
      1
      Apache 2.0