Voice-AGI MCP Server

Stateful voice-controlled AGI system combining local STT/TTS with Letta-style conversation management

Overview

Voice-AGI is an advanced MCP server that provides:

• Stateful conversations - Multi-turn dialogue with context retention

• Tool execution during voice - Call AGI functions naturally via speech

• Local STT/TTS - Cost-effective Whisper + Edge TTS (no API costs)

• Intent detection - Sophisticated NLU using local Ollama

• AGI integration - Direct control of goals, tasks, memory, and research

• Latency tracking - Performance metrics for optimization

Architecture

User Voice → Voice Pipeline (STT) → Intent Detector (Ollama)
                                            ↓
                                     Tool Registry
                                            ↓
                     ┌──────────────────────┼──────────────────────┐
                     ↓                      ↓                      ↓
            Conversation Manager    Enhanced Memory MCP    Agent Runtime MCP
                     │                      │                      │
                     └──────────────────────┴──────────────────────┘
                                            ↓
                                    AGI Orchestrator

Features

🎯 Stateful Conversation Management

• Context retention across multiple turns (last 10 turns)

• User context tracking (name, preferences, etc.)

• Conversation history stored in enhanced-memory

• Seamless multi-turn dialogue ("What was I just asking about?")

🔧 Voice-Callable AGI Tools

• search_agi_memory - Search past memories via voice

• create_goal_from_voice - "Create a goal to optimize memory"

• list_pending_tasks - "What tasks do I have?"

• trigger_consolidation - "Run memory consolidation"

• start_research - "Research transformer architectures"

• check_system_status - "How is the system doing?"

• remember_name / recall_name - User context management

• start_improvement_cycle - "Improve consolidation speed"

• decompose_goal - "Break down this goal into tasks"

• 10+ tools total, easily extensible

🧠 Intent Detection

• Local Ollama LLM (llama3.2) for sophisticated NLU

• Intent classification - Automatically routes to appropriate tools

• Parameter extraction - Extracts args from natural speech

• Context-aware - Uses conversation history for better understanding

• Fallback heuristics - Works even if Ollama unavailable

🎤 Voice Pipeline

• STT: pywhispercpp (local, Python 3.14 compatible)

• TTS: Microsoft Edge TTS (free, neural voices)

• Audio feedback: Beeps for state changes

• Latency tracking: STT, TTS, and total round-trip metrics

• Flexible: Easy to add cloud STT/TTS later

📊 Performance Metrics

• STT latency tracking (ms)

• TTS latency tracking (ms)

• Total round-trip latency

• Conversation statistics (turns, words, duration)

• Tool invocation counts

Installation

1. Install Dependencies

cd /mnt/agentic-system/mcp-servers/voice-agi-mcp
pip install -r requirements.txt

2. Ensure Prerequisites

Required:

• Python 3.10+

• edge-tts (installed via requirements.txt)

• arecord (ALSA utils): sudo dnf install alsa-utils

• Audio player: mpg123, ffplay, or vlc

• Ollama with llama3.2: ollama pull llama3.2

Optional (for STT):

• pywhispercpp: Already in requirements.txt

• Microphone access

3. Configure in Claude Code

Add to ~/.claude.json:

{
  "mcpServers": {
    "voice-agi": {
      "command": "python3",
      "args": ["/mnt/agentic-system/mcp-servers/voice-agi-mcp/src/server.py"],
      "disabled": false
    }
  }
}

4. Restart Claude Code

# Restart Claude Code to load the new MCP server

Usage

Basic Voice Chat

# From Claude Code, use the voice_chat tool:
result = voice_chat(text="Create a goal to optimize memory consolidation")

# Output:
# {
#   'response': '[Tool executed: create_goal]',
#   'tool_used': 'create_goal_from_voice',
#   'tool_result': {'goal_id': 'goal_123', ...},
#   'conversation_turns': 1
# }

Voice Conversation Loop

# Start interactive voice conversation:
result = voice_conversation_loop(max_turns=10)

# System will:
# 1. Greet you
# 2. Listen for your speech
# 3. Process intent and execute tools
# 4. Respond naturally
# 5. Continue until you say "goodbye" or max_turns reached

Listen Only

# Just transcribe speech:
result = voice_listen(duration=5)
# Returns: {'text': 'transcribed speech', 'success': True}

Speak Only

# Just speak text:
result = voice_speak(text="Hello, this is your AGI assistant")
# Returns: {'success': True, 'audio_file': '/tmp/...'}

Get Conversation Context

# View conversation history:
context = get_conversation_context()
# Returns:
# {
#   'context': 'User: ...\nAssistant: ...',
#   'summary': {'session_id': '...', 'total_turns': 5},
#   'stats': {'total_user_words': 50, ...},
#   'user_context': {'name': 'Marc'}
# }

List Voice Tools

# See all registered voice-callable tools:
tools = list_voice_tools()
# Returns: {'tools': [...], 'count': 10}

Get Performance Stats

# View latency and performance metrics:
stats = get_voice_stats()
# Returns:
# {
#   'latency': {'avg_stt_ms': 800, 'avg_tts_ms': 1500, ...},
#   'stt_available': True,
#   'tts_available': True,
#   'conversation_stats': {...},
#   'registered_tools': 10
# }

Voice-Callable Tools

Tools are automatically invoked when intent is detected in user speech.

Memory Operations

Search Memory:

User: "Search for information about transformers"
System: [Searches enhanced-memory and speaks results]

Remember User Info:

User: "My name is Marc"
System: "Got it, I'll remember your name is Marc"
...
User: "What is my name?"
System: "Your name is Marc"

Goal & Task Management

Create Goal:

User: "Create a goal to optimize memory consolidation"
System: "Goal created with ID goal_1732345678"

List Tasks:

User: "What tasks do I have?"
System: "You have 2 tasks. Task 1: Example task 1, Task 2: ..."

Decompose Goal:

User: "Break down the optimization goal into tasks"
System: "Created 5 tasks from your goal"

AGI Operations

Memory Consolidation:

User: "Run memory consolidation"
System: "Starting memory consolidation. This may take a moment."
[After processing]
System: "Consolidation complete. Found 5 patterns."

Autonomous Research:

User: "Research transformer attention mechanisms"
System: "Starting research on transformer attention mechanisms. I'll notify you when complete."

Self-Improvement:

User: "Improve consolidation speed"
System: "Starting self-improvement cycle for consolidation speed"

System Status:

User: "How is the system doing?"
System: "System is operational. 12 agents active."

Extending the System

Adding New Voice-Callable Tools

In src/server.py:

@tool_registry.register(
    intents=["your", "trigger", "keywords"],
    description="What your tool does",
    priority=8  # Higher = matched first
)
async def my_custom_tool(param: str) -> Dict[str, Any]:
    """Tool implementation"""
    try:
        # Your logic here
        result = do_something(param)

        # Speak response
        await voice_pipeline.synthesize_speech(
            f"Completed: {result}",
            play_audio=True
        )

        return result
    except Exception as e:
        logger.error(f"Error: {e}")
        return {'error': str(e)}

Customizing Intent Detection

Edit src/intent_detector.py to:

• Add new intent categories

• Adjust LLM prompts

• Tune confidence thresholds

• Add domain-specific NLU

Integrating with Other MCP Servers

Edit src/mcp_integrations.py to:

• Add new MCP client classes

• Implement actual API calls (currently stubbed)

• Configure MCP server URLs

Performance

Measured on Mac Pro 5,1 (Dual Xeon X5680, 24 threads):

Tips for Optimization:

1. Use smaller Whisper model (tiny) for faster STT

2. Pre-load Whisper model on startup

3. Use GPU if available (GTX 680 on your system)

4. Enable cloud STT/TTS for latency-critical use cases

Troubleshooting

Whisper Not Available

# Install pywhispercpp
pip install pywhispercpp

# Test:
python3 -c "from pywhispercpp.model import Model; print('✓ Whisper available')"

Edge TTS Not Working

# Install edge-tts
pip install edge-tts

# Test:
edge-tts --list-voices | grep en-IE

Ollama Not Responding

# Check Ollama is running
curl http://localhost:11434/api/generate -d '{"model":"llama3.2","prompt":"test"}'

# Pull model if needed
ollama pull llama3.2

Audio Recording Fails

# Install ALSA utils
sudo dnf install alsa-utils

# Test recording
arecord -D default -f cd -t wav -d 3 /tmp/test.wav

# List audio devices
arecord -l

No Audio Output

# Install audio player
sudo dnf install mpg123 ffmpeg

# Test playback
mpg123 /tmp/test.mp3

Architecture Details

Conversation Flow

1. User speaks → 2. STT transcribes → 3. Intent detector analyzes
                                              ↓
                                    4. Tool registry matches
                                              ↓
                                    5. Tool executes
                                              ↓
                                    6. Result spoken via TTS
                                              ↓
                                    7. Turn stored in conversation

Stateful Context

Conversation manager maintains:

• Message history (last 10 turns)

• User context (name, preferences)

• Session metadata (start time, turn count)

• Tool invocations (which tools were used)

Context is automatically:

• Passed to intent detector for better NLU

• Stored in enhanced-memory for long-term retention

• Used for multi-turn understanding

Tool Invocation

Tools are invoked when:

1. Intent confidence > 0.6

2. Intent name matches registered tool

3. Required parameters can be extracted

Parameters extracted via:

• LLM-based extraction (Ollama)

• Pattern matching (regex)

• Conversation context (previous turns)

• Defaults (if specified in tool definition)

Comparison to Letta Voice

Best of Both Worlds: This system combines Letta's stateful conversation approach with your existing local infrastructure.

Future Enhancements

Phase 4: Streaming & VAD (Planned)

• Voice Activity Detection (silero-vad)

• Streaming transcription (continuous buffer)

• Interrupt handling

• GPU acceleration for Whisper

Phase 5: Cloud Upgrade (Optional)

• Adaptive pipeline (local vs cloud based on context)

• Deepgram STT integration

• Cartesia TTS integration

• Livekit for real-time streaming

Configuration

Environment Variables

# Ollama configuration
OLLAMA_URL=http://localhost:11434
OLLAMA_MODEL=llama3.2

# Voice configuration
WHISPER_MODEL=base  # tiny, base, small, medium, large
TTS_VOICE=en-IE-EmilyNeural
TTS_RATE=+0%
TTS_VOLUME=+0%

# MCP server URLs (for integrations)
ENHANCED_MEMORY_URL=http://localhost:3000
AGENT_RUNTIME_URL=http://localhost:3001
AGI_ORCHESTRATOR_URL=http://localhost:8000

Conversation Settings

In src/server.py:

conversation_manager = ConversationManager(
    max_turns=10,  # Conversation history window
    enable_memory=True  # Store in enhanced-memory
)

Voice Pipeline Settings

voice_pipeline = VoicePipeline(
    stt_model="base",  # Whisper model size
    tts_voice="en-IE-EmilyNeural",  # TTS voice
    enable_latency_tracking=True  # Track metrics
)

API Reference

See inline docstrings in:

• src/server.py - Main MCP tools

• src/conversation_manager.py - Conversation management

• src/voice_pipeline.py - STT/TTS operations

• src/tool_registry.py - Tool registration

• src/intent_detector.py - Intent detection

• src/mcp_integrations.py - MCP client interfaces

Contributing

To add new features:

1. New voice-callable tools: Add to src/server.py with @tool_registry.register()

2. Enhanced intent detection: Update src/intent_detector.py

3. MCP integrations: Implement actual calls in src/mcp_integrations.py

4. Performance optimizations: Add VAD, streaming, GPU acceleration

5. Cloud providers: Add Deepgram/Cartesia clients

License

Part of the Mac Pro 5,1 Agentic System - see main system documentation.

Support

For issues or questions:

• Check logs: journalctl -f | grep voice-agi

• Test components individually (see Troubleshooting)

• Review AGI system documentation in /home/marc/

Voice-AGI v0.1.0 - Stateful voice control for recursive self-improving AGI systems