livekit-agent-simulator
The livekit-agent-simulator server lets you simulate AI caller interactions against a LiveKit voice agent, record detailed behavioral logs, and analyze/compare results — without touching the agent's code.
init_project— Scaffold a.agent-sim/directory (config + smoke scenario) in your target repo and add it to.gitignore.list_scenarios— List all available scenarios (from.agent-sim/scenarios/*.jsonl) with their IDs, tags, and validity status.validate_scenario— Validate a scenario file against its schema, checking for a required Persona brief and linting the PassCriteria.export_scenario— Export a parsed scenario's full details (Persona, execution parameters, PassCriteria) as JSON.run_scenario/execute_scenario— Run a simulation for a given scenario (the latter validates first), returning arun_idfor tracking.execute_scenarios— Execute multiple scenarios at once, optionally filtered by tag (e.g.smoke) or specific IDs.get_run_status— Poll a simulation run's current state (running / done / failed), turn count, and duration.get_run_log— Read the rawevents.jsonllog for a run, with filtering by event kind (supports wildcards liketool.*), turn number, source, and time offset.get_run_report— Retrieve a full report including summary, LLM judge verdict, suspicious turns, and paths to timeline/event files.compare_runs— Diff two simulation runs side-by-side across duration, turn counts, tool errors, turn-taking percentiles, and judge verdicts.list_runs— Browse run history (newest first) from SQLite, optionally filtered by scenario ID.
All tools operate on a project_root (the repo under test) and assume the target agent is already deployed.
Provides the AI simulated caller persona using Google's Gemini Live API for natural conversation testing.
Click on "Install Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@livekit-agent-simulatorrun the smoke-hello scenario"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
livekit-agent-simulator
Dial any LiveKit voice agent with an AI simulated caller — WebRTC room, inbound SIP, or outbound SIP — and keep a full forensic log.
Standalone MCP server + CLI (lks, alias: lk-sim). Black-box testing: no imports from the agent under test, no edits to its code or .env.
curl -fsSL "https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/install.sh?$(date +%s)" \
| bash -s -- --verifyInstall via coding agent (copy-paste)
Paste into Claude Code, Cursor, Codex, AmpCode, Windsurf, or any coding agent from the repo you want to test:
Install and configure livekit-agent-simulator (CLI: lks) for this project by following the instructions here:
https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/docs/guide/installation.md
Target project root is this workspace. Use absolute --root paths. Install the portable CLI if missing, run lks init, help fill .agent-sim/config.yaml from my local env or ask me for LiveKit + Gemini + agent_name, ensure .agent-sim is gitignored, run preflight, and stop before execute if the voice agent worker is not running. Do not edit agent application source outside .agent-sim/.Same idea, one line:
Install and configure livekit-agent-simulator by following: https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/docs/guide/installation.mdTL;DR
The Problem
Voice agents fail in ways unit tests never see:
Gap | What you miss |
No real caller | Scripts that never interrupt, stall, or switch language |
Chat-only evals | No room events, audio timing, or tool spans |
Manual QA calls | Not CI-reproducible, no structured PassCriteria |
Agent-coupled harnesses | Tests break when you refactor the worker |
The Solution
livekit-agent-simulator drives a Gemini Live persona from scenario JSONL over one of three transport modes (Caller.mode), observes transcripts / tools / flow / room events, and writes a timestamped report you can play back.
Surface | What you get |
| init → preflight → execute → report → web |
MCP server | Same ops for Claude Code, Cursor, Codex, … |
Transport modes |
|
Reports |
|
Judge | Optional LLM PassCriteria scoring |
Why Use lks?
Feature | What it does |
Black-box dispatch | Only needs |
5 transport modes | WebRTC · inbound SIP · outbound human pickup · outbound sim callee · agent_dials |
Scenario JSONL | Persona, Caller, Telephony, Execute, Script, PassCriteria, Dispatch |
Forensic log | Per-turn events in SQLite + |
Report player | Local web UI: audio + transcript sync |
CLI ↔ MCP parity | One |
Portable packs | Download installer; no uv/pip required for users |
Quick Example
# Install once
curl -fsSL "https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/install.sh?$(date +%s)" \
| bash -s -- --verify
# In the repo you want to test (agent worker must already be running)
lks init --root /path/to/target
# edit /path/to/target/.agent-sim/config.yaml (LiveKit + Gemini keys, agent_name)
lks preflight --root /path/to/target
lks execute smoke-hello --root /path/to/target
lks report <run-id> --root /path/to/target
lks web --root /path/to/target # Ctrl+C to stopRelated MCP server: PhoneBooth MCP Server
Design Philosophy
The agent under test is a black box.
We never import or patch target application code. Dispatch metadata is opaque JSON.Generic core, target-owned config.
Language, timezone, topics, and business strings belong in the target’s.agent-sim/— not hardcoded in the package.One ops layer for CLI and MCP.
execute_*validates then runs. No “run vs execute” forks.Forensics over vibes.
Every run produces structured events you cancompare,log, and play back.CI-friendly gates.
Hard fails on status / assert / script; optional strict judge for softer LLM scoring.
How It Works
1. Read <target>/.agent-sim/config.yaml
2. Pick SimLeg from scenario Caller.mode (webrtc_sim | inbound_sip | outbound_human_pickup | outbound_sim_callee | agent_dials)
3. Connect leg → LiveKit room(s) / SIP hairpin as needed; Gemini stays WebRTC in the sim room
4. Bridge audio; observe transcripts, tools, timing, interruptions
5. Write reports/<run-id>/ + runs.sqlite
6. Optional LLM judge vs PassCriteria Caller.mode (scenario)
┌───────────────┬────────────────┬──────────────────┬────────────────────┐
│ webrtc_sim │ inbound_sip │ outbound_human_pickup │ outbound_sim_callee│
│ room audio │ sim dials DID │ human answers → │ Gemini SIP callee │
│ │ │ Gemini colocated│ (2-room hairpin) │
└───────┬───────┴────────┬───────┴────────┬─────────┴─────────┬──────────┘
│ │ │ │
└────────────────┼────────────────┼───────────────────┘
▼
┌──────────────────────────┐
│ Gemini Live persona │
│ + LiveKit agent (black │
│ box under test) │
└────────────┬─────────────┘
│ observe
▼
reports/<run-id>/ · runs.sqlite · judgeMode details and config: docs/telephony.md. Templates: inbound-caller-sim, outbound-human-pickup, outbound-callee-sim.
How lks Compares
Approach | Real room | AI caller | Forensic log | MCP | Black-box |
Manual phone QA | ✅ | ❌ | ❌ | ❌ | ✅ |
Unit / mock STT | ❌ | ❌ | Partial | ❌ | ❌ |
In-repo agent tests | ⚠️ | ⚠️ | Varies | ❌ | Often coupled |
lks | ✅ LiveKit | ✅ Gemini Live | ✅ Full | ✅ | ✅ |
When to use lks:
Regression suites for LiveKit voice agents
Agent-driven CI / coding-agent workflows (MCP)
Debugging turn-taking, tools, and silence without reading agent source
When it might not be ideal:
Pure text chatbots with no LiveKit room
Offline environments without LiveKit + Gemini API access
Installation
Quick install (recommended)
Download only — no uv/pip/build on your machine. CI ships a portable pack (embedded Python + deps + report player).
# macOS / Linux
curl -fsSL "https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/install.sh?$(date +%s)" \
| bash -s -- --verify# Windows PowerShell
irm "https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/install.ps1" -OutFile "$env:TEMP\lk-sim-install.ps1"
powershell -NoProfile -ExecutionPolicy Bypass -File "$env:TEMP\lk-sim-install.ps1" -VerifyAlso available from a release asset:
curl -fsSL "https://github.com/quangdang46/livekit-agent-simulator/releases/download/v0.1.0/install.sh" \
| bash -s -- --verifyFlag | Purpose |
| Checksum verification |
| Pin release tag |
| Skip MCP registration into coding tools |
| Remove install |
By default the installer registers the MCP server livekit-agent-simulator (lks mcp) into detected tools: Claude Code, Cursor, Cline, Windsurf, VS Code Copilot, Gemini CLI, Amazon Q, OpenCode, Codex, Warp.
Agent-oriented install playbook (long form): docs/guide/installation.md
Raw URL for paste into agents:https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/docs/guide/installation.md
From source (maintainers / contributors)
git clone https://github.com/quangdang46/livekit-agent-simulator.git
cd livekit-agent-simulator
uv sync --extra dev
uv run lks --helpRequires Python 3.10–3.13.
Web UI (maintainers)
Users never build this — CI packs web/dist into the wheel as web_static. Source: web/.
pnpm --dir web install
pnpm --dir web build # → web/dist/ (attached by Hatch on uv build)
pnpm --dir web dev # HMR; proxy /api + /runs → lks web :8765See web/README.md.
Quick Start
# Agent worker must be running and registered with LiveKit
lks guide
lks init --root /path/to/target
# fill .agent-sim/config.yaml
lks preflight --root /path/to/target
lks scenario-init smoke-hello --root /path/to/target # if needed
lks validate smoke-hello --root /path/to/target
lks execute smoke-hello --root /path/to/target
lks runs --root /path/to/target
lks report <run-id> --root /path/to/target
lks web --root /path/to/targetMinimal scenario (smoke-hello)
{"apiVersion":"agent-sim/v1","kind":"Scenario","metadata":{"id":"smoke-hello","locale":"en-US","tags":["smoke"]}}
{"kind":"Persona","spec":{"name":"Alex","brief":"First-time caller; confirm you reached the right place, then end politely.","goals":["Hear the agent","Say you will call back"],"style":"polite, brief"}}
{"kind":"Execute","spec":{"max_turns":2,"timeout_s":90,"first_speaker":"user"}}
{"kind":"PassCriteria","spec":{"criteria":["The agent responded to the caller","The agent responded in the caller's language"]}}Optional multi-judge PassCriteria: judges[] + mode (all | majority | any). Assert highlights (tool_order, constraint_respected, recovery/latency): lks guide.
Full-line // comments in scaffolded JSONL are guides — runtime ignores them.
Configuration
Target-only data lives under <target>/.agent-sim/ (gitignored). Created by init.
Section | Required | Purpose |
| yes |
|
| yes | Server API credentials |
| yes | Must match worker dispatch name |
| no | Default opaque JSON string for all runs |
| yes | Gemini key for sim caller (+ judge) |
| no | Defaults: flash-live, Puck, |
| no | If set + PassCriteria → post-run LLM judge |
| no (default | Local stereo WAV (L=sim, R=agent); no Egress |
| no | Empty = all topics |
| no | Map data payloads → tool start/end/error |
See template: templates/config.yaml. Consumer-specific wiring: docs/portability.md.
Commands
CLI and MCP share the same public ops (ops.py). Prefer execute (validate then run).
CLI | MCP tool | Purpose |
|
| Scaffold |
|
| Setup/ops guide (markdown) |
|
| Local report player |
|
| Config + LiveKit connectivity |
|
| List |
|
| Verify plugins |
|
| Built-in + local PCM cues |
|
| Schema + lint |
|
| Parsed scenario JSON |
|
| Scaffold JSONL with |
|
| Validate then run one scenario |
|
| Batch (ids / tag) |
|
| In-memory scenario dict |
|
| SQLite run status |
|
| Filtered |
|
| Summary + verdict + paths |
|
| Diff two runs; |
|
| Run history |
| — | Start MCP server (stdio) |
lks execute smoke-hello --root /path/to/target
lks execute-all --tag smoke --root /path/to/target
lks log <run-id> --root /path/to/target
lks compare <run-a> <run-b> --root /path/to/target
lks compare <baseline> <candidate> --baseline --root /path/to/target
lks web --port 8765 --root /path/to/targetEvery MCP tool needs project_root except guide.
MCP config examples
Installer writes this when tools are detected. Manual Cursor:
{
"mcpServers": {
"livekit-agent-simulator": {
"command": "lks",
"args": ["mcp"],
"env": {}
}
}
}Dev checkout (package not installed globally):
{
"mcpServers": {
"livekit-agent-simulator": {
"command": "uv",
"args": ["run", "--directory", "/abs/path/livekit-agent-simulator", "lks", "mcp"]
}
}
}Equivalent one-shot entry: lks-mcp (alias: lk-sim-mcp; same process as lks mcp / lk-sim mcp).
Architecture
src/livekit_agent_simulator/
├── cli.py / mcp_server.py # thin surfaces
├── ops.py # shared public ops
├── run_orchestrator.py # room lifecycle + run
├── scenario.py # JSONL parse / validate
├── config.py # .agent-sim/config.yaml
├── preflight.py
├── asserts.py / suite.py # CI gates
├── gemini/ # Live caller + judge
├── livekit/ # room, dispatch, observe
├── audio/ · script/ · plugins/
└── web/ # report player serverLayer | Role |
Target | Config, scenarios, reports, local plugins/cues |
Package | Scaffold defaults + built-in cues |
LiveKit | Room, dispatch, data topics, transcription |
Gemini Live | Simulated caller voice + optional judge |
CI / Release
Workflow | Trigger | What it does |
PR / push → | web UI build, | |
tag | pytest → wheel → portable packs (win/linux/mac) → GitHub Release |
# Local check
uv sync --extra dev
pnpm --dir web build
uv run pytest -q
# Release (pre-1.0 may force-retag a single 0.1.0)
git tag v0.1.0
git push origin v0.1.0Troubleshooting
preflight fails connectivity
lks preflight --root /path/to/target
# Confirm livekit.url / api_key / api_secret and that the project is reachable.
# Skip API check while editing config:
lks preflight --no-connectivity --root /path/to/targetAgent never joins the room
Worker process must be running and registered with the same
livekit.agent_name.Increase
livekit.agent_join_timeout_msif cold start is slow.Check dispatch metadata is valid JSON string if your worker requires it.
Gemini / simulator auth errors
Set simulator.google_api_key in .agent-sim/config.yaml. The sim caller uses Gemini Live (gemini-3.1-flash-live-preview by default).
No audio in report player
With observe.record_audio enabled (default true): reports/<run-id>/conversation.wav
lks web --root /path/to/targetMCP tools not listed
lks mcp # must be what the host launches
# or reinstall without --no-mcp
curl -fsSL "https://raw.githubusercontent.com/quangdang46/livekit-agent-simulator/main/install.sh?$(date +%s)" \
| bash -s -- --verifyScenario validation errors
lks validate my-case --root /path/to/target
lks scenario-init my-case --root /path/to/target # fresh scaffold with // guidesLimitations
What lks Doesn't Do (Yet)
Not an agent framework — it tests agents; it does not implement business tools
Not offline-first — needs LiveKit + Gemini (or configured backends)
Not a load generator — one simulated caller per run (batch via
execute-all)
Known Limitations
Capability | Current state | Notes |
Black-box dispatch | ✅ | Opaque metadata only |
Multi-caller rooms | ❌ | Single sim participant |
Non-Gemini caller backends | ⚠️ | Gemini Live is the supported path |
Pixel-perfect ASR scoring | ❌ | Use PassCriteria + judge / asserts |
Secrets in config | ⚠️ Paste in gitignored YAML | Do not commit |
FAQ
Does it modify my agent repo?
Only scaffolds .agent-sim/ (gitignored). It does not edit agent source.
CLI vs MCP — which should I use?
Same ops. Use CLI in terminals/CI; MCP inside coding agents. Prefer execute_* over ad-hoc run paths.
How do I pass project-specific dispatch fields?
livekit.dispatch_metadata or scenario Dispatch.spec.metadata as an opaque JSON string. Core does not parse consumer keys. See docs/portability.md.
Can I assert on tool calls?
Yes — Assert.spec.tools, tool_order (required tool.start subsequence), observe.tool_event_patterns, Script/assert plugins, and/or PassCriteria + judge. See docs/plugins.md and lks guide.
Where are reports stored?
<target>/.agent-sim/reports/<run-id>/ plus runs.sqlite under .agent-sim/.
Is the report player separate?
No — lks web serves the prebuilt player from the install pack. Maintainers build from web/.
Docs
Doc | When |
Rules for AI agents working on this package | |
First end-to-end run | |
Consumer dispatch / observe setup | |
Verify plugins + Python API | |
SIP modes + outbound_sim_callee preflight | |
Barge / backchannel / noise authoring | |
| On-demand setup/ops guide (Assert, compare --baseline, PassCriteria) |
About Contributions
Please don't take this the wrong way, but I do not accept outside contributions for any of my projects. I simply don't have the mental bandwidth to review anything, and it's my name on the thing, so I'm responsible for any problems it causes; thus, the risk-reward is highly asymmetric from my perspective. I'd also have to worry about other "stakeholders," which seems unwise for tools I mostly make for myself for free. Feel free to submit issues, and even PRs if you want to illustrate a proposed fix, but know I won't merge them directly. Instead, I'll have Claude or Codex review submissions via gh and independently decide whether and how to address them. Bug reports in particular are welcome. Sorry if this offends, but I want to avoid wasted time and hurt feelings. I understand this isn't in sync with the prevailing open-source ethos that seeks community contributions, but it's the only way I can move at this velocity and keep my sanity.
License
Black-box LiveKit agent tests. Real rooms. Forensic reports.
Maintenance
Resources
Unclaimed servers have limited discoverability.
Looking for Admin?
If you are the server author, to access and configure the admin panel.
Latest Blog Posts
- Your AI Chatbot Just Exposed Your CEO's Salary to an InternBy Om-Shree-0709 on .Agent IdentityMCP SecurityOAuth Delegation
- Why MCP Servers Need Execution Sandboxing (And Why Your Current Stack Isn't Enough)By Om-Shree-0709 on .Agentic AiPrompt InjectionWebAssembly
MCP directory API
We provide all the information about MCP servers via our MCP API.
curl -X GET 'https://glama.ai/api/mcp/v1/servers/quangdang46/livekit_agent_simulator'
If you have feedback or need assistance with the MCP directory API, please join our Discord server