Run Simulation
run_simulationLaunch a multi-turn conversation evaluation of your AI agent, using a simulated user and scenario to generate realistic interactions with quality checks, and retrieve results asynchronously.
Instructions
Run a multi-turn conversation evaluation of your AI agent.
Combines a Target (the system under test), a Driver (the simulated user),
and a Scenario (the test cases) to generate realistic multi-turn conversations
and evaluate them with quality checks.
Returns promptly so the call never times out on long runs. Short runs that
finish within the buffer window return ``status: "finished"`` with results
ready; longer runs return ``status: "running"`` with the ``test_run_id``,
``app_link``, and an ``estimated_runtime`` — the run continues to completion
on its own. In both cases, poll get_test_run_results with the returned
test_run_id for scores, and get_conversation_transcript for transcripts.
To rerun a previous simulation — keeping its configuration but changing one or
more parameters — pass based_on_run_id with the original run's ID and supply
only the values you want to override. If scenario_name or target_name are
omitted and based_on_run_id is provided, they will be resolved from the
original run.
For custom_endpoint Targets: an exception raised during the run (for
example the endpoint erroring mid-conversation) FAILS the run — it is
reported as a failed simulation, not silently skipped.
**Voice augmentations** — for voice Targets, the `augmentation` parameter
applies realistic acoustic and conversational effects. Six top-level keys:
`cap`, `directed_speech`, `secondary_speaker`, `backchannel`, `barge_in`,
plus the composable `noise`. **Composition rule**: at most one non-noise
strategy may be active, optionally combined with `noise`. Augmentations
apply only to voice Targets — calls against generation or custom_endpoint
Targets with an augmentation block are rejected. Field-level errors
(out-of-range probability, missing required field, swapped offsets, unknown
strategy) are returned by the MCP before any backend call.
Strategy required / optional fields (numeric ranges in brackets):
- cap: probability [0.0, 1.0] required. pause_ms [0, 10000] optional.
- directed_speech: probability [0.0, 1.0] required. lpf_cutoff_hz (>0),
gain_db [-40.0, 0.0], sample_rate (>0), prompt, reverb_preset optional.
- secondary_speaker: probability [0.0, 1.0] AND secondary_voice (non-empty
string) required. inter_speaker_pause_ms [0, 5000], lpf_cutoff_hz (>0),
gain_db [-40.0, 0.0], sample_rate (>0), secondary_prompt,
secondary_voice_instructions, secondary_reverb_preset optional.
- backchannel: utterance (non-empty string) required. probability
[0.0, 1.0], min_offset_ms (>=0), max_offset_ms (>= min_offset_ms),
seed optional.
- barge_in: prompt (non-empty string) required. probability [0.0, 1.0],
min_offset_ms (>=0), max_offset_ms (>= min_offset_ms), seed optional.
- noise: noise_profile (non-empty string) AND noise_snr_db (number)
required. seed optional.
For copy-paste examples and the full reference, call
`get_templates(["voice_augmentations"])`.
Args:
name: Human-readable name for this simulation run.
scenario_name: Name of the scenario to use. Required unless based_on_run_id
is provided and the original run's scenario can be resolved.
target_name: Name of the target to evaluate. Required unless based_on_run_id
is provided and the original run's target can be resolved.
driver_name: Name of the driver persona. If omitted, the project default
driver is used.
checks: List of check names to apply (from list_checks). Pick from
the list_checks category matching the task and modality —
voice-specific categories for voice simulations, categories
outside them for either modality; never chat-only checks for
audio (or vice versa). Every simulation runs with at least one
check: when omitted or empty, the benign code-based "latency"
performance check is applied automatically and the response
discloses the substitution via `default_check_applied`.
Supplied checks are used unchanged.
repeats: Number of times to run each scenario row, default 1.
max_turns: Maximum conversation turns per simulation, default 5.
first_turn: Who speaks first — 'target' or 'driver', default 'target'.
based_on_run_id: ID of a previous simulation run to reuse parameters from.
Explicitly supplied values override the original run's parameters.
augmentation: (voice Targets only) Voice augmentation block. See the
"Voice augmentations" section above for keys and ranges. An empty
dict is treated as no augmentation.
turn_transition_time: Milliseconds of pause between turns. Forwarded to
the backend as-is; SDK default (1000) is used when omitted.
silence_timeout_ms: The target reply timeout — how patient Okareo
is before indicating that the target can't respond. Do NOT set
or change this value unless the user specifically directs it;
it should be 10000 ms in nearly all cases. It exists to
accommodate untuned targets with very long tool calls, during
which the Driver waits patiently. It does NOT change how fast
Okareo responds, and lowering it does not speed anything up —
a slow simulation is not a reason to change it. Forwarded to
the backend; backend default is used when omitted.
checks_at_every_turn: When True, checks are evaluated per turn (not
only at end of run).
stop_check: Early-stop config: `{"check_name": str, "stop_on": <value>}`.
The run halts as soon as the named check returns `stop_on`.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | ||
| checks | No | ||
| repeats | No | ||
| max_turns | No | ||
| first_turn | No | target | |
| stop_check | No | ||
| driver_name | No | ||
| target_name | No | ||
| augmentation | No | ||
| scenario_name | No | ||
| based_on_run_id | No | ||
| silence_timeout_ms | No | ||
| checks_at_every_turn | No | ||
| turn_transition_time | No |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| result | Yes |