Voice Mode

# Streaming Audio Playback Specification ## Overview This specification describes implementing progressive/streaming audio playback in voicemode to reduce latency by playing audio as it arrives rather than waiting for the complete file. ## Current State - Audio is fully downloaded before playback begins - Uses `with_streaming_response.create()` but reads entire response - Adds 3-6 seconds of latency for TTS generation - All formats are buffered completely before playing ## Proposed Implementation ### 1. Streaming Architecture ```python async def stream_tts_audio(text: str, client: AsyncOpenAI, **params): """Stream TTS audio with progressive playback.""" # Audio pipeline components audio_queue = asyncio.Queue(maxsize=10) # Buffer chunks playback_started = asyncio.Event() playback_complete = asyncio.Event() # Download task async def download_chunks(): async with client.audio.speech.with_streaming_response.create( **params ) as response: async for chunk in response.iter_bytes(chunk_size=4096): await audio_queue.put(chunk) await audio_queue.put(None) # Signal end # Playback task async def play_chunks(): # Initialize audio stream stream = await create_audio_stream(params['response_format']) while True: chunk = await audio_queue.get() if chunk is None: break # Decode and play chunk await stream.play_chunk(chunk) if not playback_started.is_set(): playback_started.set() await stream.close() playback_complete.set() # Run both tasks concurrently download_task = asyncio.create_task(download_chunks()) playback_task = asyncio.create_task(play_chunks()) # Wait for playback to start await playback_started.wait() # Continue both tasks await asyncio.gather(download_task, playback_task) ``` ### 2. Format-Specific Considerations #### PCM (Recommended for Streaming) - **Advantage**: No decoding needed, direct playback - **Disadvantage**: Large bandwidth requirement - **Use case**: Local TTS with minimal latency #### Opus - **Container**: OGG with Opus codec - **Advantages**: - Designed for low-latency streaming - Small chunk sizes (2.5-60ms frames) - Built-in error resilience - **Implementation**: Use `opuslib` or `pyogg` for decoding - **Critical Limitation**: Doesn't work for streaming - must be fully buffered before playback - **Recommendation**: Use PCM for streaming TTS, Opus only for file storage #### MP3 - **Challenge**: Frame boundaries may not align with chunks - **Solution**: Buffer until valid frame header found - **Typical frame size**: ~400-1400 bytes ### 3. Audio Playback Backend Options #### Option 1: PyAudio Streaming (Cross-platform) ```python import pyaudio class PyAudioStream: def __init__(self, format, channels, rate): self.pa = pyaudio.PyAudio() self.stream = self.pa.open( format=format, channels=channels, rate=rate, output=True, stream_callback=self.callback, frames_per_buffer=1024 ) self.buffer = queue.Queue() def callback(self, in_data, frame_count, time_info, status): # Pull from buffer and play data = self.get_audio_data(frame_count) return (data, pyaudio.paContinue) ``` #### Option 2: Sounddevice Streaming ```python import sounddevice as sd import numpy as np class SoundDeviceStream: def __init__(self, samplerate, channels): self.buffer = asyncio.Queue() self.stream = sd.OutputStream( samplerate=samplerate, channels=channels, callback=self.audio_callback, blocksize=2048 ) async def play_chunk(self, audio_data): await self.buffer.put(audio_data) ``` ### 4. Buffering Strategy - **Initial buffer**: 100-200ms before starting playback - **Adaptive buffering**: Adjust based on network conditions - **Underrun handling**: Pause and rebuffer if needed - **Maximum buffer**: 1-2 seconds to prevent memory issues ### 5. Implementation Phases #### Phase 1: Basic Streaming (PCM/WAV) - Implement streaming for uncompressed formats - Test latency improvements - Establish buffering parameters #### Phase 2: Compressed Format Support - Implement MP3 frame boundary detection - Handle format-specific quirks #### Phase 3: Adaptive Streaming - Monitor buffer health - Adjust chunk sizes dynamically - Implement quality fallback ### 6. Configuration ```bash # Enable streaming playback VOICEMODE_STREAMING_ENABLED=true # Initial buffer size (ms) VOICEMODE_STREAM_BUFFER_MS=150 # Chunk size for download VOICEMODE_STREAM_CHUNK_SIZE=4096 # Maximum buffer size (seconds) VOICEMODE_STREAM_MAX_BUFFER=2.0 ``` ### 7. Error Handling - **Network interruption**: Pause playback, attempt reconnection - **Decoder errors**: Skip corrupted chunks, log warnings - **Buffer underrun**: Insert silence, increase buffer size - **Format incompatibility**: Fall back to buffered playback ### 8. Metrics and Monitoring Track these metrics for optimization: - Time to first audio (TTFA) - Buffer underrun count - Average buffer level - Network throughput - Chunk decode time ### 9. Testing Strategy 1. **Unit tests**: Mock streaming responses, test decoders 2. **Integration tests**: Real API calls with streaming 3. **Network simulation**: Test with various latencies/bandwidth 4. **Format testing**: Verify each format streams correctly 5. **Stress testing**: Multiple concurrent streams ### 10. Benefits - **Reduced latency**: Start playback within 150-200ms - **Better UX**: User hears response beginning immediately - **Memory efficient**: No need to buffer entire response - **Scalable**: Handle longer responses without memory issues ### 11. Risks and Mitigations | Risk | Mitigation | |------|------------| | Audio glitches | Adequate buffering, quality monitoring | | Format incompatibility | Fallback to buffered mode | | CPU overhead | Efficient decoders, worker threads | | Network issues | Adaptive buffering, reconnection logic | ### 12. Future Enhancements - **WebRTC integration**: For ultra-low latency - **Spatial audio**: Stream multi-channel audio - **Adaptive bitrate**: Adjust quality based on network - **Caching**: Stream from cache for repeated phrases