# MCP Virtual User

A synthetic human in a Docker container. Has ears, a mouth, eyes, and hands.

## What is this?

A fully self-contained environment that simulates a real user interacting with web apps via voice and browser. It has:

- **Virtual microphone** — MCP can inject audio (TTS or raw) that any app thinks is coming from a hardware mic
- **Virtual speakers** — MCP can capture and transcribe whatever audio the OS plays back
- **Real browser** — Chromium with Playwright control, persistent login sessions
- **Real display** — Xvfb + VNC for debugging (watch what's happening live)
- **MCP interface** — Everything exposed as tools via Streamable HTTP

## Use Cases

- **Test ChatGPT voice mode** — inject "What's the weather?" into the mic, capture ChatGPT's spoken response, transcribe it, assert on it
- **Test Gemini Live** — same flow against Google's voice AI
- **Test our Mobile Mesh UI** — full end-to-end voice conversation testing against our own app
- **Any voice-enabled web app** — if it uses the browser's mic/speaker, we can test it

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│  Docker Container                                            │
│                                                             │
│  ┌─────────────────────────────────────────────────────┐    │
│  │  MCP Server (port 8360)                              │    │
│  │                                                     │    │
│  │  Audio Tools:                                       │    │
│  │    tts_to_mic(text)     → Piper TTS → virtual mic   │    │
│  │    transcribe_speakers() → parec → Whisper STT      │    │
│  │    inject_audio(b64)    → raw audio → virtual mic   │    │
│  │    capture_audio(secs)  → raw audio from speakers   │    │
│  │    wait_for_speech()    → detect + transcribe       │    │
│  │                                                     │    │
│  │  Browser Tools:                                     │    │
│  │    browser_navigate, click, type, screenshot, etc.  │    │
│  │    browser_grant_mic_permission(origin)             │    │
│  │                                                     │    │
│  │  Screen Tools:                                      │    │
│  │    screen_screenshot, screen_size, vnc_url          │    │
│  └──────────┬──────────────────────────┬───────────────┘    │
│             │                          │                     │
│  ┌──────────▼──────────┐  ┌───────────▼───────────────┐    │
│  │  PulseAudio         │  │  Playwright + Chromium    │    │
│  │                     │  │                           │    │
│  │  virtual_mic ◀──────│──│── browser reads as mic    │    │
│  │  (pipe-source)      │  │                           │    │
│  │                     │  │  browser plays audio ──▶  │    │
│  │  virtual_speaker ───│──│── captured via .monitor   │    │
│  │  (null-sink)        │  │                           │    │
│  └─────────────────────┘  └───────────────────────────┘    │
│                                                             │
│  ┌─────────────────────┐  ┌───────────────────────────┐    │
│  │  Xvfb :99           │  │  x11vnc + noVNC          │    │
│  │  1920x1080x24       │  │  port 5900 / 6080        │    │
│  └─────────────────────┘  └───────────────────────────┘    │
└─────────────────────────────────────────────────────────────┘
```

## Quick Start

```bash
# Build and start
docker compose up -d --build

# Watch the virtual display (open in your browser)
open http://localhost:6080/vnc.html?autoconnect=true

# Test the MCP server
curl http://localhost:8360/mcp -X POST \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"health","arguments":{}}}'

# Run smoke tests
bash test-smoke.sh
```

## First-Time Setup (One-Time)

```bash
# 1. Build and start
docker compose up -d --build

# 2. Open VNC in your browser to see the desktop
open http://localhost:6080/vnc.html?autoconnect=true

# 3. In the VNC window, Chromium is running. Log into:
#    - https://chat.openai.com (ChatGPT)
#    - https://gemini.google.com (Gemini)

# 4. Export cookies to DragonsKeep so they persist
bash scripts/export-browser-cookies.sh

# 5. Done! Future container starts auto-inject the stored sessions.
```

## Running Tests

```bash
# Infrastructure tests (no login required)
cd tests && pip install -r requirements.txt
pytest test_audio_roundtrip.py test_browser.py -v

# ChatGPT conversation tests (requires login)
pytest test_conversations.py -v -m chatgpt --timeout=120

# Gemini conversation tests
pytest test_conversations.py -v -m gemini --timeout=120

# Our Mesh UI conversation tests
pytest test_conversations.py -v -m meshui --timeout=120

# All conversation tests
pytest test_conversations.py -v --timeout=120
```

## Example: Test ChatGPT Voice Mode

```python
# From any MCP client (Kiro, our mesh agent, etc.)

# 1. Navigate to ChatGPT
browser_navigate("https://chat.openai.com")

# 2. Grant mic permission
browser_grant_mic_permission("https://chat.openai.com")

# 3. Click the voice mode button
browser_click("[data-testid='voice-mode-button']")

# 4. Speak a question (injected into the virtual mic)
tts_to_mic("What is the capital of France?")

# 5. Wait for ChatGPT to respond and transcribe what it says
response = wait_for_speech(timeout_seconds=15)
# response == "The capital of France is Paris."

# 6. Assert
assert "Paris" in response
```

## Session Management

Login sessions persist in `./data/browser-profile/` (mounted volume).

For ChatGPT/Gemini auth tokens, use DragonsKeep:
1. Store cookies in DragonsKeep as `chatgpt-session` / `gemini-session`
2. On container start, inject them into the browser profile

Or: log in manually once via VNC (http://localhost:6080), session persists.

## Ports

| Port | Service |
|------|---------|
| 8360 | MCP Server (Streamable HTTP) |
| 6080 | noVNC (browser-based VNC viewer) |
| 5900 | VNC direct |

## MCP Tools

### Audio
| Tool | Description |
|------|-------------|
| `tts_to_mic` | Synthesize text → inject as microphone input |
| `transcribe_speakers` | Capture speaker output → transcribe to text |
| `inject_audio` | Push raw audio bytes into the virtual mic |
| `capture_audio` | Record raw audio from the virtual speakers |
| `wait_for_speech` | Detect speech on speakers, wait for it to finish, transcribe |

### Browser
| Tool | Description |
|------|-------------|
| `browser_navigate` | Go to a URL |
| `browser_click` | Click an element |
| `browser_type` | Type into an input |
| `browser_press_key` | Press a keyboard key |
| `browser_screenshot` | Take a page screenshot |
| `browser_get_text` | Get page text content |
| `browser_evaluate` | Execute JavaScript |
| `browser_wait_for_text` | Wait for text to appear |
| `browser_url` | Get current URL |
| `browser_grant_mic_permission` | Allow mic access for an origin |

### Screen
| Tool | Description |
|------|-------------|
| `screen_screenshot` | Full desktop screenshot |
| `screen_size` | Get display resolution |
| `vnc_url` | Get the live VNC viewer URL |