# LocalVoiceMode

**Local voice interface with Character Skills** - Self-contained voice chat system.

Uses **Parakeet TDT 0.6B** (NVIDIA) for fast GPU speech recognition, **Pocket TTS** (Kyutai) for natural text-to-speech. Auto-detects **LM Studio**, **OpenRouter**, or **OpenAI** as the LLM backend.

## Features

- **Parakeet TDT ASR** - NVIDIA's fast speech recognition (GPU accelerated via ONNX)
- **Pocket TTS** - Kyutai's natural-sounding text-to-speech with voice cloning
- **Smart Turn Detection** - Knows when you're done speaking, not just detecting silence
- **Auto-Provider Detection** - Automatically finds LM Studio, or falls back to OpenRouter/OpenAI
- **Modern Rich UI** - Beautiful terminal interface with audio visualization
- **Character Skills** - Load different personalities with custom voices
- **MCP Integration** - Works with Claude Code and other MCP-enabled tools

## Quick Start

### 1. Clone and Setup

```batch
git clone https://github.com/your-username/localvoicemode.git
cd localvoicemode
setup.bat
```

This creates a virtual environment and installs all dependencies.

### 2. HuggingFace Login (Required)

Pocket TTS requires accepting the model license:

```batch
.venv\Scripts\huggingface-cli.exe login
```

Then accept the license at: https://huggingface.co/kyutai/pocket-tts

### 3. Configure LLM Provider

**Option A: LM Studio (Recommended for local)**
1. Open LM Studio
2. Load your preferred model
3. Start the local server (default: `http://localhost:1234`)

**Option B: OpenRouter**
```batch
set OPENROUTER_API_KEY=your-key-here
```
Get your key at: https://openrouter.ai/keys

**Option C: OpenAI**
```batch
set OPENAI_API_KEY=your-key-here
```

### 4. Run Voice Chat

```batch
REM Default assistant
VoiceChat.bat

REM With Hermione character
VoiceChat.bat hermione

REM Push-to-talk mode
VoiceChat.bat hermione ptt
```

## Provider Detection

LocalVoiceMode automatically detects available providers in this order:

1. **LM Studio** - Scans ports 1234, 1235, 1236, 8080, 5000
2. **OpenRouter** - Uses `OPENROUTER_API_KEY` environment variable
3. **OpenAI** - Uses `OPENAI_API_KEY` environment variable

Force a specific provider with `VOICE_PROVIDER=openrouter` (or `lm_studio`, `openai`).

## Directory Structure

```
localvoicemode/
├── voice_client.py        # Main voice client entry point
├── mcp_server.py          # MCP server for AI assistant integration
├── requirements.txt       # Python dependencies
├── setup.bat              # Setup script (run first!)
├── VoiceChat.bat          # Launch script
├── start_voicemode.bat    # MCP server launcher
│
├── src/localvoicemode/    # Core package
│   ├── audio/             # Audio recording
│   ├── speech/            # ASR, TTS, VAD, filters
│   ├── llm/               # Provider management
│   ├── skills/            # Skill loading
│   └── state/             # State machines, config
│
├── skills/                # Character skills
│   ├── assistant-default/ # Default assistant
│   └── hermione-companion/
│       ├── SKILL.md       # Character definition
│       ├── references/    # Lore files
│       └── scripts/       # Helper scripts
│
└── voice_references/      # Custom voice files (.wav)
```

## Skills System

Skills define character personalities, system prompts, and optional knowledge.

### List Available Skills

```batch
.venv\Scripts\python.exe voice_client.py --list-skills
```

### Create a New Skill

1. Create directory: `skills/my-skill/`
2. Create `SKILL.md`:

```yaml
---
id: my-skill
name: My Character
display_name: "My Character"
description: Brief description
metadata:
  greeting: "Hello! How can I help?"
---

# My Character

## System Prompt

You are My Character. [Full instructions here...]
```

3. Add optional files:
   - `reference.wav` - Voice clone source (10s of clear speech)
   - `avatar.png` - Character image
   - `references/` - Knowledge markdown files

### Voice Cloning

Pocket TTS supports voice cloning from reference audio.

**Requirements:**
- WAV format (16-bit PCM)
- ~10 seconds of clean speech
- Clear recording, minimal background noise

Place the file at:
- `skills/my-skill/reference.wav` (per-skill), or
- `voice_references/my-skill.wav` (global)

## Voice Modes

### VAD Mode (default)

Voice Activity Detection with Smart Turn - automatically detects when you're done speaking.

```batch
VoiceChat.bat hermione
```

### PTT Mode

Push-to-Talk - hold Space to record, release to send.

```batch
VoiceChat.bat hermione ptt
```

## Configuration

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `VOICE_API_URL` | `http://localhost:1234/v1` | OpenAI-compatible API URL |
| `VOICE_API_KEY` | (none) | API key for the provider |
| `VOICE_MODEL` | (auto) | Model name to use |
| `VOICE_PROVIDER` | (auto) | Force provider: lm_studio, openrouter, openai |
| `OPENROUTER_API_KEY` | (none) | OpenRouter API key |
| `OPENAI_API_KEY` | (none) | OpenAI API key |
| `VOICE_TTS_VOICE` | `alba` | Default TTS voice |
| `VOICE_DEVICE` | `cuda` | ASR device: cuda (GPU) or cpu |
| `VOICE_SMART_TURN_THRESHOLD` | `0.5` | Turn completion threshold (0.0-1.0) |

### Command Line Options

```
python voice_client.py [options]

Options:
  --skill, -s SKILL      Load a character skill
  --list-skills, -l      List available skills
  --list-providers       List available LLM providers
  --provider, -p PROV    Force provider: lm_studio, openrouter, openai
  --mode, -m MODE        Input mode: vad, ptt, or type
  --device DEVICE        ASR device: cuda or cpu
  --api-url URL          OpenAI-compatible API URL
  --api-key KEY          API key for the provider
  --model MODEL          Model name to use
  --headless             Run without UI (for MCP integration)
```

## MCP Integration

LocalVoiceMode includes an MCP server for integration with Claude Code and other MCP-enabled tools.

### Start MCP Server

```batch
start_voicemode.bat
```

### Available Tools

- `speak(text)` - Speak text aloud (TTS)
- `listen()` - Listen for speech (STT)
- `converse(text)` - Speak and listen for response
- `start_voice(skill)` - Start voice chat with a character
- `stop_voice()` - Stop voice chat
- `voice_status()` - Check if voice mode is running
- `list_voices()` - List available characters
- `provider_status()` - Show available providers
- `set_speech_mode(mode)` - Set verbosity: roleplay, coder, minimal, silent
- `get_speech_mode()` - Get current speech mode

### Slash Commands

These slash commands are available in Claude Code and compatible AI assistants:

| Command | Description |
|---------|-------------|
| `/speak <text>` | TTS only - speak text aloud |
| `/listen` | STT only - transcribe speech to text |
| `/tts-only` | Mode: Claude speaks, you type |
| `/stt-only` | Mode: You speak, Claude responds in text |
| `/voice-roleplay` | Full expressive speech output |
| `/voice-coder` | Summaries & completions only |
| `/voice` | Speak one message via voice |
| `/voice-on` | Start continuous voice mode |
| `/voice-off` | Stop voice mode |
| `/voice-typing` | You type, Claude speaks (hold RIGHT SHIFT to speak) |

### Speech Modes

Control how much Claude speaks:

| Mode | Description |
|------|-------------|
| `roleplay` | Full expressive output - speaks everything naturally (default) |
| `coder` | Summaries only - task completions, errors, questions |
| `minimal` | Very terse - only critical announcements |
| `silent` | No speech - text only |

Switch modes with `/voice-roleplay`, `/voice-coder`, or the `set_speech_mode()` tool.

### Voice Commands While Running

- Say **"stop"** or **"goodbye"** to end
- Say **"change voice"** to switch characters

## GPU Support

Parakeet TDT uses ONNX Runtime with GPU acceleration:

1. **TensorRT** (best performance) - Auto-detected if installed
2. **CUDA** (good performance) - Requires CUDA/cuDNN
3. **CPU** (fallback) - Always available

Check GPU status:
```batch
.venv\Scripts\python.exe -c "import onnxruntime as ort; print(ort.get_available_providers())"
```

## Troubleshooting

### No audio detected
- Check microphone permissions
- Verify default audio device: `python -c "import sounddevice; print(sounddevice.query_devices())"`

### Pocket TTS not working
- Accept license: https://huggingface.co/kyutai/pocket-tts
- Login: `.venv\Scripts\huggingface-cli.exe login`

### LM Studio connection failed
- Verify LM Studio server is running
- Check URL: default is `http://localhost:1234`
- Ensure a model is loaded

### OpenRouter/OpenAI not working
- Verify API key is set in `.env` or environment
- Check `python voice_client.py --list-providers` to see detected providers

### GPU/CUDA not working
- Ensure NVIDIA drivers are installed
- Install CUDA Toolkit 12.x
- Reinstall: `pip uninstall onnxruntime onnxruntime-gpu && pip install onnxruntime-gpu[cuda,cudnn]`

## Credits

- **Parakeet TDT**: [NVIDIA NeMo](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2) - Apache 2.0
- **Pocket TTS**: [Kyutai](https://huggingface.co/kyutai/pocket-tts) - CC-BY-4.0
- **Smart Turn**: [Livekit](https://github.com/livekit/smart-turn) - Apache 2.0
- **Silero VAD**: [Silero](https://github.com/snakers4/silero-vad) - MIT