claude-voice-mcp

Bidirectional local voice for Claude Code on Apple Silicon, built on mlx-audio. Talk to Claude, and Claude talks back — automatically, guaranteed by a Claude Code Stop hook rather than by Claude remembering to call a speak() tool. Runs 100% locally and free by default (Whisper + Kokoro), with an optional ElevenLabs backend for more realistic voices.

Contents: Two ways to use this · Commands · Why a Stop hook · Setup · Scope · Configuration · Architecture

Two ways to use this

Option 1 — Claude Code's native dictation + /talkback for replies. Claude Code has its own built-in push-to-talk dictation (tap Space in the chat box) that types your speech into the input box — that's a separate system from this project, not something we built. Use it for input, and run /talkback on so this MCP speaks Claude's replies back via Kokoro. Our own mic/STT pipeline (listen, /talk) is never used in this mode — only the TTS half. Simplest option if you're happy with Claude Code's own dictation and just want spoken replies on top of it.

Option 2 — /talk for a fully local, hands-free conversation. /talk arms hands_free mode and starts recording through our own local pipeline (Whisper in, Kokoro out): it records until you stop talking, transcribes it, Claude responds, the Stop hook speaks the reply, then it automatically starts recording again — a continuous loop with no typing and no dictation button, until you say "stop listening", go quiet past hands_free_idle_seconds, or run /talk off. Entirely local end-to-end.

/talk only arms listening. If you've ever turned spoken replies off with /talkback off, /talk alone won't bring them back — run /talkback on too, or you'll be transcribed and continued but never hear a reply. Both default to on, so a fresh install gets the full experience automatically.

The two options mix freely — e.g. native dictation most of the time, /talk when you want your hands off the keyboard.

Related MCP server: Voice Loop MCP

Commands

Say "stop listening" any time to end hands-free mode by voice instead of typing /talk off.

Why a Stop hook instead of a speak() tool?

Tool-call-based auto-speak (the common approach) only works if the model chooses to call the tool after every reply — it can forget, get distracted, or skip it under load. This project instead wires a Stop hook (hooks/speak_on_stop.py) that Claude Code invokes after every turn, independent of the MCP server and independent of Claude's cooperation. It reads Claude's own final message for the turn, strips code/markdown/paths, and speaks it (a short summary by default, or the whole thing with /talkback full). Everything about this is tunable live via the voice_config MCP tool or a config file, no restart required.

The same hook is also what makes hands-free mode work: when armed, it listens again after speaking and feeds what you say back in via its decision: "block" output, so Claude Code continues the conversation without you retyping anything.

Other capabilities

• stop_speaking() — barge-in: interrupt playback mid-sentence.

• list_voices() — same as /voice, callable directly.

• voice_config() — get/set any setting live from Claude's console (see Configuration), persisted to ~/.claude-voice-mcp/config.json (or a project-local .voice-mcp.json).

• vocabulary() / /vocab — teach Whisper words it mishears (app names, jargon, rare terms), two layers deep: words are fed to it as an initial_prompt on every transcription (a standard Whisper biasing technique), and a fuzzy-correction pass (vocab_correct.py) catches near-misses the prompt alone didn't fix (e.g. "bido app" heard for "bedouapp") and swaps in the exact taught spelling. Neither layer is a hard guarantee, but together they measurably help with made-up names and jargon Whisper otherwise guesses wrong.

• Hallucination guard (stt_guard.py) — Whisper occasionally hallucinates a repeating phrase from silence/noise (a known failure mode). Detected and trimmed automatically before it reaches the conversation.

• Echo guard (stt_guard.is_echo_of) — in hands-free mode, the mic can pick up Claude's own TTS reply through the speakers and transcribe it back as if it were new user input, which without a check would let Claude reply to its own echo indefinitely. The hook compares what was just spoken against what the mic just heard and silently discards a match; a short settle delay before re-listening reduces how often this happens at all.

• Duplicate-hook guard — if the Stop hook ends up registered in more than one scope at once (e.g. both this project's .claude/settings.json and a global ~/.claude/settings.json, which both fire when working inside this repo), Claude Code runs every registered copy, and Claude would audibly speak the same reply twice. The hook atomically claims each turn's prompt_id so only the first invocation for a given turn does anything; every duplicate silently no-ops.

• Warm-model daemon — the Stop hook is a fresh process every turn; without this it would reload Kokoro/Whisper from scratch each time (5+ seconds). server.py keeps a background daemon with both models warm so the hook stays fast.

• Optional ElevenLabs backend for more realistic voices: set ELEVENLABS_API_KEY and voice_config set tts_backend elevenlabs. If the key is missing or a call fails for any reason, it silently falls back to local Kokoro — voice never breaks because of the cloud.

Requirements

• Apple Silicon Mac (M1 or later), macOS

• Python 3.11+, uv

• Working microphone and speakers

• ffmpeg (optional, only needed for MP3/FLAC handling)

Setup on a new machine

1. Clone this repo somewhere permanent (its path gets baked into config below, so pick a final location, e.g. ~/tools/claude-voice-mcp):

   git clone git@github.com:Amaldeep98/Claude-realtime-voice-mcp.git
   cd Claude-realtime-voice-mcp

2. Run setup:

   ./scripts/setup.sh

   Installs uv if missing (via ~/.local/bin), pins Python 3.12 (spaCy, one of Kokoro's text-processing dependencies, doesn't yet have wheels for newer Pythons), syncs the environment, and pre-downloads the default models (~2-3GB): Kokoro-82M (TTS) and Whisper large-v3-turbo (STT).

3. The Stop hook is already configured — this repo ships its own .claude/settings.json with the hook wired up using Claude Code's ${CLAUDE_PROJECT_DIR} placeholder, which resolves to wherever you cloned this repo, so nothing needs editing:

   {
     "hooks": {
       "Stop": [
         {
           "hooks": [
             {
               "type": "command",
               "command": "uv --project ${CLAUDE_PROJECT_DIR} run python ${CLAUDE_PROJECT_DIR}/hooks/speak_on_stop.py",
               "timeout": 120
             }
           ]
         }
       ]
     }
   }

   The 120s timeout matters: the hook speaks and then listens in hands-free mode, which can legitimately take over a minute — a shorter timeout silently kills it mid-listen with no error shown. If you start Claude Code fresh from a directory that already has this file (i.e. you cd'd into the clone before launching claude), it's picked up automatically; if you were already mid-session when the clone appeared, run /hooks once to reload.

   ${CLAUDE_PROJECT_DIR} only resolves correctly for this project, though — if you register the hook globally instead (see Scope below), it needs a real absolute path, since a global hook fires from inside whatever project you're currently in, not from this one.

4. Register the MCP server so Claude Code can see the listen/speak/etc. tools — see Scope below for project-only vs. everywhere.

Scope: project-only vs. available everywhere

Three independent pieces, each defaulting to this project only, each with an "everywhere" option:

For everywhere-by-default, do the "global" option for all three of MCP server, slash commands, and Stop hook. To opt in per-project instead, add matching .mcp.json + .claude/commands/ + .claude/settings.json entries to each project you want it in — e.g. for another project's .mcp.json:

{
  "mcpServers": {
    "voice": {
      "type": "stdio",
      "command": "uv",
      "args": ["--directory", "/path/to/claude-voice-mcp", "run", "server.py"]
    }
  }
}

Configuration

All settings live in ~/.claude-voice-mcp/config.json and can be changed live via the voice_config MCP tool (voice_config(action="set", key="...", value="...")) — no restart required.

Architecture

voice_mcp/
  config.py        # shared config, read by both the MCP server and the standalone hook
  audio_io.py       # mic capture + VAD, playback, cue tones, macOS notifications
  sanitize.py       # strip markdown/code/urls/paths before any TTS call
  summarizer.py     # turn a raw assistant turn into a short spoken summary
  stt_guard.py      # detect/trim Whisper hallucination (repeat loops) on noise/silence
  vocab_correct.py  # fuzzy-correct near-misses of taught vocabulary words (see /vocab)
  stt/              # whisper_backend.py (default), voxtral_backend.py (opt-in)
  tts/              # kokoro_backend.py (default), elevenlabs_backend.py (optional)
  tools.py          # tool implementations shared by server.py, the daemon, and the hook's fallback
  daemon.py         # background Unix-socket server (in server.py's process) keeping models warm
  ipc_client.py     # lightweight client the hook uses to reach the daemon, no heavy imports
server.py           # FastMCP entrypoint: listen, speak, stop_speaking, list_voices, voice_config;
                     # also starts the daemon in a background thread
hooks/speak_on_stop.py  # Stop hook: guaranteed auto-speak + hands-free, independent of the MCP server
                         # (talks to the daemon for speed, falls back to loading models directly)