| list_checksA | List available quality checks, grouped by category. Returns checks (both built-in and custom) available in your Okareo
account, organized into `checks_by_category` using the platform's
`__category:<Category>` tags; checks with no category appear under
`uncategorized`. Select checks from the category matching your task
AND modality: voice-specific categories (e.g. voice/audio quality)
apply to voice simulations, while checks outside voice-specific
categories are generally useful for both chat and voice. A check
carrying multiple categories appears under each of them.
Each check has a name, description, and output_data_type.
output_data_type uses the server vocabulary: "bool" is a pass/fail
check and "int" is a scored check — these correspond to output_type
"pass_fail" and "score" in create_or_update_check and generate_check.
Use these check names with run_test to evaluate model quality.
Args:
limit: Maximum number of checks to return (default 20), applied to
the total before grouping. Use 0 for no limit.
all_versions: When false (default), returns only the latest version of
each check. When true, returns the full version history of every
check, each entry annotated with its version number.
|
| run_testA | Submit a quality test that evaluates a model against a scenario using checks. Returns promptly so the call never times out on long runs. Short runs return
``status: "finished"`` with results ready; longer runs return
``status: "running"`` with the ``test_run_id`` and ``app_link`` — the run
continues to completion on its own. In both cases, poll get_test_run_results
with the returned test_run_id to retrieve scores.
Args:
scenario_name: Name of the scenario to evaluate against.
model_name: Name of the registered model to evaluate.
checks: List of check names to apply (e.g., ["coherence", "relevance"]).
Use list_checks to discover available checks and pick from the
category matching the task and modality — do not use
voice-specific checks for text evaluations (or vice versa);
checks outside voice-specific categories suit both.
name: Optional human-readable name for this test run.
type: Type of evaluation. Defaults to NL_GENERATION. Valid values:
NL_GENERATION, INFORMATION_RETRIEVAL, MULTI_CLASS_CLASSIFICATION,
INVARIANT, MULTI_TURN, AGENT_EVAL.
|
| list_test_runsA | List past test runs in the project. Returns test run names, IDs, timestamps, status, and summary scores,
sorted by most recent first. Defaults to the 10 most recent runs.
Optionally filter by model name, scenario name, or type.
For simulation runs (type MULTI_TURN), use get_test_run_results with the
returned test_run_id to retrieve full conversation transcripts and per-turn
check scores.
Args:
model_name: Optional filter — only show test runs using this model.
scenario_name: Optional filter — only show test runs using this scenario.
limit: Maximum number of runs to return, sorted by most recent first.
Defaults to 10. Set to 0 to return all runs.
simulation_only: When True, return only MULTI_TURN simulation runs.
Useful for browsing past simulation results without NL_GENERATION or
other test run types appearing in the list.
|
| get_test_run_resultsA | Load the results of a specific test run. Look up by test run ID (UUID) or by name (returns the most recent run
matching that name). Returns aggregate metrics and per-row check scores.
By default, conversation transcripts (model_input/model_result) are
excluded to keep responses concise. Set include_transcripts=True to
include full transcripts. Use get_conversation_transcript to inspect
a single conversation's transcript without loading all of them.
Supports pagination via limit and offset for large result sets.
Args:
test_run_id: The UUID of the test run. Takes precedence over name.
name: The name of the test run. Returns the most recent match.
include_transcripts: Include full model_input and model_result in
each data point. Defaults to False (scores only). Set True for
full conversation transcripts.
limit: Maximum number of data points to return. 0 (default) returns
all data points. Use with offset for pagination.
offset: Number of data points to skip. Defaults to 0.
|
| get_conversation_transcriptA | Retrieve the full conversation transcript for a single data point. Use this after get_test_run_results to drill into a specific
conversation. Provide either scenario_index (1-based, from the
scores summary) or test_id (UUID) to identify the conversation.
Returns the complete message transcript (model_input), final
output (model_result), per-turn check scores (metric_value),
and the scenario seed data.
Args:
test_run_id: The UUID of the test run.
scenario_index: 1-based index of the conversation within
the test run. Visible in get_test_run_results output.
test_id: UUID of the specific data point. Alternative to
scenario_index.
|
| reevaluate_test_runA | Re-score a completed test run against a set of checks. Re-runs checks against an already-finished test run without re-executing
the original model or simulation, and without changing the original
run's results. Useful after a check definition changed, or to score an
existing run against additional checks.
Args:
test_run_id: UUID or name of a completed test run.
checks: Optional list of check names (or IDs) to score against.
When omitted, the run's existing checks are re-run.
|
| save_scenarioA | Save a named scenario for use in quality tests. Provide EXACTLY ONE dataset source: `content`, `file_path`, or
`rows`.
Prefer `file_path` for local .jsonl files — the server reads the
file directly, so no rows pass through the assistant's context.
When passing rows through the assistant instead, keep the dataset
UNDER 2,000 rows (`content` with the file's text, or `rows` for
tiny datasets). For 2,000 rows or more, always use `file_path` or
upload directly to Okareo via the web app, SDK, or CLI, to avoid
unnecessary token cost.
If a scenario with the same name already exists, the existing
scenario is returned (idempotent). Scenarios are immutable after
creation — use create_scenario_version to create updated versions.
Args:
name: A unique name for the scenario.
content: Raw JSONL text (one JSON object with 'input' and
'result' per line). Only for datasets under 2,000 rows.
file_path: Path to a local .jsonl file. Preferred — works for
any size.
rows: List of data rows, each with 'input' (any type) and
'result' (any type). Use for small scenarios (< 20 rows).
tags: Optional list of tags for categorizing the scenario.
|
| list_scenariosA | List scenarios in the project, most recent first. Returns scenario names, IDs, tags, row counts, and creation dates.
Use this to discover existing scenarios before running a test.
Args:
limit: Maximum number of scenarios to return (default 20).
Set to 0 to return all scenarios.
|
| get_scenarioA | Read a scenario's metadata and all data rows. Look up by name or scenario ID. Returns scenario details and all
input/result data rows.
Args:
name: Name of the scenario to retrieve.
scenario_id: ID of the scenario to retrieve. Takes precedence over name.
|
| create_scenario_versionA | Create a new version of an existing scenario with updated data. Automatically determines the next version number (e.g., 'my-test-v2',
'my-test-v3'). The original scenario is treated as version 1.
Args:
base_name: Name of the original scenario to create a version of.
rows: List of data rows for the new version, each with 'input' and 'result'.
|
| preview_delete_scenarioA | Preview what will be deleted before removing a scenario. Shows the scenario details and count of related test runs that will
also be deleted. Use delete_scenario to confirm deletion after reviewing.
Args:
name: Name of the scenario to preview deletion for.
scenario_id: ID of the scenario. Takes precedence over name.
|
| delete_scenarioA | Permanently delete a scenario and all related test data. Both scenario_id and name are required. Use preview_delete_scenario first
to see what will be deleted before confirming.
Args:
scenario_id: The ID of the scenario to delete (from preview_delete_scenario).
name: The name of the scenario to delete.
|
| list_available_llmsA | Browse available LLMs from the Okareo registry. Returns all LLMs that can be used when registering a generation model
for testing. Each entry has a name, display name, and provider. Use a
model_name from this list when calling register_generation_model.
|
| register_generation_modelA | Register a generation model for testing by selecting an LLM from the registry. Creates a generation model (Model Under Test) that points to a specific LLM
(e.g., 'azure/gpt-4o-mini'). Use list_available_llms to see available
LLMs. The registered generation model can then be used with run_test.
Args:
name: A human-readable name for this generation model (e.g., 'my-chatbot').
model_name: The LLM from the registry (from list_available_llms).
|
| list_generation_modelsA | Browse all registered generation models in the project. Returns generation model names, IDs, target LLM configurations, and
creation timestamps. Use this to see what generation models are
available for testing. |
| get_generation_modelA | Read detailed information about a registered generation model. Returns the generation model's target LLM configuration, tags, creation
time, and any warnings (e.g., if the target LLM has been deprecated).
Args:
name: Name of the registered generation model.
|
| update_generation_modelA | Change the LLM that a registered generation model points to. Updates the generation model to use a different LLM from the registry.
Use list_available_llms to see available LLMs.
Args:
name: Name of the registered generation model to update.
model_name: The new LLM from the registry.
|
| delete_generation_modelA | Remove a registered generation model and all its related test data. Permanently deletes the generation model and cascades to associated
test runs and test data points. This cannot be undone.
Args:
name: Name of the registered generation model to delete.
|
| create_or_update_targetA | Create or update a Target — the AI system you want to evaluate in a simulation. Calling create_or_update_target with the same name as an existing Target will
**fully replace** its configuration — caller must re-specify all desired fields.
Supported types: 'generation' (foundation model), 'custom_endpoint' (your own
REST API), and 'voice' (voice-based targets reached by phone via Twilio or
over SIP).
**Cloning workflow**: this tool accepts the same key structure that `get_target`
returns, so you can read an existing Target, change `name`, swap in real values
for any field whose value is `"***REDACTED***"`, and pass the result here as
kwargs. Calls that still contain the redaction sentinel are rejected with an
error naming each offending path; the sentinel is never forwarded to the backend.
Args:
name: Unique name for this target.
type: Target type — 'generation', 'custom_endpoint', or 'voice'.
model_id: (generation targets) Foundation model ID, e.g. 'gpt-4o-mini'.
temperature: (generation targets) Response randomness, default 0.
system_prompt_template: (generation targets) System instructions; mustache
syntax supported, e.g. '{scenario_input}'.
user_prompt_template: (generation targets) User prompt template.
dialog_template: (generation targets) Dialog formatting template.
tools: (generation targets) Tool definitions for function calling.
next_message_params: (custom_endpoint) Nested HTTP config for each
conversation turn. Required keys: 'url', 'method'. Optional:
'headers', 'body', 'status_code', 'response_message_path',
'response_session_id_path', 'response_tool_calls_path'.
All response path values MUST use dot-path notation starting with
'response.' — e.g., 'response.message', 'response.choices[0].message.content',
'response.choices[0].tool_calls'. Never use bare property names.
For SSE/streaming endpoints, include a 'streaming' object with:
- 'stop': array of stop conditions (OR semantics — any match ends
the stream). Each has 'value' (required) and optional 'path'
(dot-path into JSON chunk). Without 'path', matches raw SSE data.
- 'select': array of select conditions (AND semantics — all must
match for a chunk's content to be extracted). Each requires
'path' and 'value'.
When streaming, set response_message_path to the chunk field
(e.g., 'response.choices[0].delta.content').
start_session_params: (custom_endpoint, optional) Nested HTTP config to
initialise a session. Required key: 'url'. Optional: 'method',
'headers', 'body', 'status_code', 'response_session_id_path'
(dot-path starting with 'response.', e.g. 'response.id'),
'response_message_path'. Supports 'streaming' object (same
structure as next_message_params.streaming).
end_session_params: (custom_endpoint, optional) Nested HTTP config to
close a session after the last turn.
auth_params: (custom_endpoint, optional) Token-based authorization config.
Required keys when provided: 'url', 'method', 'response_access_token_path'
(dot-path starting with 'response.', e.g. 'response.access_token').
Optional: 'headers', 'body', 'status_code'.
sensitive_fields: (custom_endpoint, optional) List of dot-path strings for
secret fields within auth_params (e.g., 'auth_params.body.client_id').
The MCP auto-generates entries for top-level auth_params keys; use this
for deeper paths. To remove auth from an existing target, call
create_or_update_target again without auth_params.
max_parallel_requests: (custom_endpoint, twilio) Concurrency limit. This is
the same setting the Okareo web UI labels "max concurrency".
edge_type: (voice targets) How Okareo reaches the voice agent —
'twilio' (dial a phone number) or 'sip' (call a SIP URI).
to_phone_number: (voice twilio) Destination phone number (required).
account_sid: (voice twilio, custom only) Twilio account SID. If provided,
auth_token and from_phone_number are also required (all-or-nothing).
Omit for generic Twilio targets using Okareo's managed integration.
auth_token: (voice twilio, custom only) Twilio auth token. Required with
account_sid and from_phone_number.
from_phone_number: (voice twilio, custom only) Caller phone number. Required
with account_sid and auth_token.
sip_uri: (voice sip) Destination SIP URI (required), e.g.
"sip:agent@your-domain.example.com". Use this to test any voice
agent reachable over SIP — for example one fronted by Daily,
Vapi, LiveKit, or a SIP trunk.
sip_username: (voice sip, optional) SIP authentication username.
sip_password: (voice sip, optional) SIP authentication password
(stored as a sensitive field).
|
| get_targetA | Check the current configuration of a test target. Retrieves a Target by name. Works for all target types (Generation,
Custom Endpoint, and Voice).
For **custom_endpoint** Targets, the response is a flat envelope whose
keys mirror the kwargs accepted by `create_or_update_target`, so a
copilot can read the result, swap in a new name + secrets, and feed it
back to create to clone the Target. Fields the backend keeps secret
(those listed in `sensitive_fields`) appear with the literal value
`"***REDACTED***"` — these MUST be replaced with real values before
calling `create_or_update_target`, which rejects payloads still
containing the sentinel.
The `max_parallel_requests` field on custom_endpoint Targets is the
same setting the Okareo web UI labels "max concurrency".
For generation and voice Targets, the response shape is unchanged
(kept stable for existing callers).
Args:
name: Name of the target to retrieve.
|
| list_targetsA | Browse all simulation targets available in this project. Returns all simulation targets (voice and custom_endpoint types)
created via create_or_update_target. Does not include generation models
registered via register_generation_model — use list_generation_models
for those.
|
| delete_targetA | Remove a simulation target and all its related test data. Permanently deletes the target and cascades to associated
test runs and test data points. This cannot be undone.
Args:
name: Name of the target to delete.
|
| create_or_update_driverA | Define a simulated user persona that will interact with your target. Creates or updates a Driver by name (upsert). Author ONLY the core
persona sections in prompt_template, in this order:
- `## Persona` — who the simulated user is (static character).
- `## Scenario Details` — contains the scenario reference
(`{scenario_input}` or a specific path like
`{scenario_input.objectives}`), placed immediately before
Objectives. This is how each scenario row's data reaches the
conversation.
- `## Objectives` — WHAT the driver is trying to accomplish, written
from the driver's goal (not from scenario variables).
- `## Soft Tactics` — HOW the driver probes, escalates, and stops.
Do NOT author Hard Rules, a Turn-End Checklist, or Conversation
Behavior sections: the MCP automatically appends the platform's
canonical versions of those blocks (including the language rule
matching `language`) — the same blocks the Okareo UI appends to
generated drivers. Any caller-authored variant of these sections is
replaced by the canonical text, and repeated updates never duplicate
the blocks.
For voice agents, configure how the simulated user speaks with `voice`,
`voice_profile`, `voice_instructions`, and `language`. Call
list_driver_voices first to discover valid voice and profile values.
Args:
name: Unique name for this driver.
prompt_template: The core persona prompt (Persona, Scenario
Details, Objectives, Soft Tactics — see above). Hard Rules and
Conversation Behavior are appended automatically.
model_id: Foundation model to power the driver (defaults to project default).
temperature: Response randomness, default 0.6.
voice_instructions: Free-text speaking instructions for voice simulations
(tone, pace, accent). Not validated against the voice catalog.
voice_profile: Voice profile name for voice simulations. Validated
against the catalog from list_driver_voices.
voice: Voice identifier for voice simulations. Validated against the
catalog from list_driver_voices.
language: Language the driver responds in, as the bare ISO code
the voice catalog serves (e.g. "en", "es", "ja"); regional
variants like "fr-CA" are accepted when their base code
matches the voice's language. When a `voice` is set and
language is omitted, it is derived from that voice's catalog
language (disclosed as `language_derived_from_voice` in the
response); a value conflicting with the voice's language is
rejected. Also drives the appended Hard Rules language rule.
|
| get_driverA | Retrieve a driver persona you've already configured. Retrieves a Driver by name, returning its full configuration including the
persona prompt.
Args:
name: Name of the driver to retrieve.
|
| list_driversA | See what driver personas are available in this project. Returns all Drivers with their names, IDs, model, and temperature. |
| list_driver_voicesA | Discover the voices, voice profiles, and languages available for
configuring voice-capable drivers. Call this before create_or_update_driver when building a voice agent
simulation, so you can pass valid `voice`, `voice_profile`, and
`language` values.
Each entry in `voices` carries selection metadata — use it to pick
the voice:
- `language`: bare ISO code (e.g. "en", "es", "ja"). The driver's
`language` is derived from the selected voice's language when
omitted, and must not conflict with it.
- `accent`: free-text accent label present on many voices (e.g.
"British", "Mexican", "Parisian", "Southern US"). To satisfy an
accent request, select a voice whose `accent` matches — writing
accent instructions into `voice_instructions` does NOT change the
TTS voice.
- `gender`: e.g. "feminine" / "masculine".
`voice_profiles` are emotion/delivery presets (happy, angry,
sarcastic, ...) — they shape affect, not accent or language.
`languages` lists the distinct voice languages available.
|
| run_simulationA | Run a multi-turn conversation evaluation of your AI agent. Combines a Target (the system under test), a Driver (the simulated user),
and a Scenario (the test cases) to generate realistic multi-turn conversations
and evaluate them with quality checks.
Returns promptly so the call never times out on long runs. Short runs that
finish within the buffer window return ``status: "finished"`` with results
ready; longer runs return ``status: "running"`` with the ``test_run_id``,
``app_link``, and an ``estimated_runtime`` — the run continues to completion
on its own. In both cases, poll get_test_run_results with the returned
test_run_id for scores, and get_conversation_transcript for transcripts.
To rerun a previous simulation — keeping its configuration but changing one or
more parameters — pass based_on_run_id with the original run's ID and supply
only the values you want to override. If scenario_name or target_name are
omitted and based_on_run_id is provided, they will be resolved from the
original run.
For custom_endpoint Targets: an exception raised during the run (for
example the endpoint erroring mid-conversation) FAILS the run — it is
reported as a failed simulation, not silently skipped.
**Voice augmentations** — for voice Targets, the `augmentation` parameter
applies realistic acoustic and conversational effects. Six top-level keys:
`cap`, `directed_speech`, `secondary_speaker`, `backchannel`, `barge_in`,
plus the composable `noise`. **Composition rule**: at most one non-noise
strategy may be active, optionally combined with `noise`. Augmentations
apply only to voice Targets — calls against generation or custom_endpoint
Targets with an augmentation block are rejected. Field-level errors
(out-of-range probability, missing required field, swapped offsets, unknown
strategy) are returned by the MCP before any backend call.
Strategy required / optional fields (numeric ranges in brackets):
- cap: probability [0.0, 1.0] required. pause_ms [0, 10000] optional.
- directed_speech: probability [0.0, 1.0] required. lpf_cutoff_hz (>0),
gain_db [-40.0, 0.0], sample_rate (>0), prompt, reverb_preset optional.
- secondary_speaker: probability [0.0, 1.0] AND secondary_voice (non-empty
string) required. inter_speaker_pause_ms [0, 5000], lpf_cutoff_hz (>0),
gain_db [-40.0, 0.0], sample_rate (>0), secondary_prompt,
secondary_voice_instructions, secondary_reverb_preset optional.
- backchannel: utterance (non-empty string) required. probability
[0.0, 1.0], min_offset_ms (>=0), max_offset_ms (>= min_offset_ms),
seed optional.
- barge_in: prompt (non-empty string) required. probability [0.0, 1.0],
min_offset_ms (>=0), max_offset_ms (>= min_offset_ms), seed optional.
- noise: noise_profile (non-empty string) AND noise_snr_db (number)
required. seed optional.
For copy-paste examples and the full reference, call
`get_templates(["voice_augmentations"])`.
Args:
name: Human-readable name for this simulation run.
scenario_name: Name of the scenario to use. Required unless based_on_run_id
is provided and the original run's scenario can be resolved.
target_name: Name of the target to evaluate. Required unless based_on_run_id
is provided and the original run's target can be resolved.
driver_name: Name of the driver persona. If omitted, the project default
driver is used.
checks: List of check names to apply (from list_checks). Pick from
the list_checks category matching the task and modality —
voice-specific categories for voice simulations, categories
outside them for either modality; never chat-only checks for
audio (or vice versa). Every simulation runs with at least one
check: when omitted or empty, the benign code-based "latency"
performance check is applied automatically and the response
discloses the substitution via `default_check_applied`.
Supplied checks are used unchanged.
repeats: Number of times to run each scenario row, default 1.
max_turns: Maximum conversation turns per simulation, default 5.
first_turn: Who speaks first — 'target' or 'driver', default 'target'.
based_on_run_id: ID of a previous simulation run to reuse parameters from.
Explicitly supplied values override the original run's parameters.
augmentation: (voice Targets only) Voice augmentation block. See the
"Voice augmentations" section above for keys and ranges. An empty
dict is treated as no augmentation.
turn_transition_time: Milliseconds of pause between turns. Forwarded to
the backend as-is; SDK default (1000) is used when omitted.
silence_timeout_ms: The target reply timeout — how patient Okareo
is before indicating that the target can't respond. Do NOT set
or change this value unless the user specifically directs it;
it should be 10000 ms in nearly all cases. It exists to
accommodate untuned targets with very long tool calls, during
which the Driver waits patiently. It does NOT change how fast
Okareo responds, and lowering it does not speed anything up —
a slow simulation is not a reason to change it. Forwarded to
the backend; backend default is used when omitted.
checks_at_every_turn: When True, checks are evaluated per turn (not
only at end of run).
stop_check: Early-stop config: `{"check_name": str, "stop_on": <value>}`.
The run halts as soon as the named check returns `stop_on`.
|
| list_simulationsA | List past simulation runs in the project. Returns simulation run names, IDs, timestamps, and status, sorted by
most recent first. Defaults to the 10 most recent runs in summary mode.
Use detail_level="detailed" to include model_metrics and additional
fields (limit is capped to 5 in detailed mode to prevent overflow).
Use get_test_run_results with the returned test_run_id to retrieve
per-row scores (transcripts excluded by default). Then use
get_conversation_transcript with a scenario_index to inspect
individual conversation transcripts.
Args:
target_name: Optional filter — only show simulation runs using
this target.
scenario_name: Optional filter — only show simulation runs using
this scenario.
limit: Maximum number of runs to return, sorted by most recent
first. Defaults to 10. Set to 0 to return all runs.
detail_level: "summary" (default) returns compact results without
model_metrics. "detailed" returns full results with metrics
(limit capped to 5).
|
| create_or_update_checkA | Create or update a quality check by name (upsert). Supports model-based, code-based, and audio checks. Saving to an existing name creates a new version of that check (see
get_check's "available_versions"). Before writing a prompt_template or
code_contents from scratch, fetch a worked example with get_templates:
"boolean_check_prompt", "score_check_prompt", "analysis_check_prompt",
or "check_code".
Args:
name: Unique name for the check.
description: What the check evaluates.
check_type: "model" (an LLM judge driven by prompt_template) or
"code" (a deterministic Python class in code_contents).
output_type: "pass_fail" (boolean verdict), "score" (numeric, e.g.
a 1-5 rubric), or "analysis" (free-form qualitative feedback;
only valid with check_type="model"). For check_type="code" the
server infers pass_fail vs score from the value evaluate()
returns (bool vs int/float) — output_type is used only to
validate the request, not sent to the server. Note: list_checks
and get_check report this as output_data_type in the server
vocabulary, where "bool" means pass_fail and "int" means score.
prompt_template: Required when check_type="model". The judge
prompt. Inject the runtime data the judge needs with these
placeholders:
- {model_output}: the model output being evaluated. In a
multi-turn conversation this is ONLY the final assistant
message, not the full conversation.
- {scenario_input}: the scenario input / source text.
- {scenario_result}: the reference/expected output.
- {model_input}: what was sent to the model (prompt or
messages).
- {message_history}: the full multi-turn conversation — the
model_input messages plus the assistant's model_output. Use
this when the check must judge the whole conversation.
- {tool_calls}: the tool/function calls the model just made.
- {tools}: the tool definitions/schema available to the model.
- {model_output_metadata}: metadata attached to the most
recent model output.
- {simulation_message_history}: full conversation history
reconstructed from trace metadata. Only populated for traced
(ingested) conversations; for simulations and evaluations
use {message_history}.
The legacy {generation} placeholder is deprecated — use
{model_output} instead.
code_contents: Required when check_type="code" (output_type
"pass_fail" or "score" only). Python source defining
`class Check(CodeBasedCheck)` with a
`@staticmethod def evaluate(...) -> CheckResponse` method.
Start from `from okareo.checks import CodeBasedCheck,
CheckResponse`. evaluate() may declare any subset of these
parameters: model_output, scenario_input, scenario_result,
metadata, model_input. Return CheckResponse(score=...,
explanation=...) where score is a bool for pass_fail or an
int/float for score. See get_templates("check_code") for
complete examples.
is_audio: Set to true for audio/voice evaluation. Only valid with
check_type="model".
tags: Optional list of string tags to organize the check. Tags are
stored with the check and returned by get_check.
|
| generate_checkA | Generate a check from a natural language description. Uses AI to create the prompt template (model checks) or Python code (code checks), then saves the check. Use this when you only have a description of what to evaluate. When
you already know the exact prompt template or Python code the check
should use, call create_or_update_check directly instead. The
generated prompt/code is returned in the response — review it and
refine with create_or_update_check if needed.
Args:
name: Name for the generated check.
description: Natural language description of what to evaluate
(e.g., "check if the response is toxic"). The more specific
the description, the better the generated check.
output_type: "pass_fail" (boolean verdict), "score" (numeric), or
"analysis" (free-form qualitative feedback; model checks only).
check_type: "model" (LLM judge) or "code" (deterministic Python).
requires_scenario_input: Set true when the evaluation must compare
the output against the scenario input. The generated check
will reference {scenario_input} and only works on runs whose
scenarios provide it.
requires_scenario_result: Set true when the evaluation must
compare the output against the expected result. The generated
check will reference {scenario_result} and only works on runs
whose scenarios provide it.
|
| get_checkA | Retrieve the full configuration of a check by name, including its prompt template or code contents. Args:
name: Name of the check to retrieve.
version: Optional check version number to pin. Omit (or leave null)
for the most recent version. The response always lists every
available version under "available_versions".
|
| delete_checkA | Permanently delete a check by name. Args:
name: Name of the check to delete.
|
| get_docsA | Query the Okareo documentation system for information about Okareo primitives and workflows. Use this tool when the agent or user needs to understand how Okareo
concepts work — Scenarios, Checks, Targets, Drivers, Evaluations,
and Simulations.
Two modes are available:
- 'conceptual': Detailed technical documentation for agent reasoning.
Default top_k=5 (returns up to 5 documentation entries).
- 'user_legible': Plain-language explanations for human users.
Default top_k=3 (returns up to 3 documentation entries).
If the Okareo documentation service is unavailable (e.g. air-gapped
environment), the tool returns a helpful error suggesting get_templates
as a fallback.
Args:
query: The question to ask the Okareo documentation system. Be
specific — e.g., 'How do Checks and Evaluations work together?'
or 'What is a Driver persona?'
mode: Documentation mode — 'conceptual' or 'user_legible'.
top_k: Number of documentation entries to return. Defaults to 5
for conceptual mode, 3 for user_legible mode. Maximum 10.
|
| get_templatesA | Retrieve prompt templates for common Okareo patterns. Returns starter templates for building Okareo test components. These
templates are served as static content from the MCP — no network calls
required. Always available, even in air-gapped environments.
Available templates:
- basic_scenario: Template for creating a basic Okareo test scenario
- boolean_check_prompt: Template for a pass/fail (boolean) check prompt
- score_check_prompt: Template for a scored check prompt
- check_code: Template for a code-based check (Python function)
- target_validate_check_prompt: Template for validating target output
- driver_prompt: Template for a Driver persona prompt
- driver_voice_extension_prompt: Template for voice interaction extensions
- analysis_check_prompt: Template for an analysis check (qualitative feedback)
Args:
template_name: Template identifier to retrieve. Omit to get a
lightweight listing of all available templates (names and
descriptions only). Provide a template_name to get the full
template content. Valid values: basic_scenario,
boolean_check_prompt, score_check_prompt, check_code,
target_validate_check_prompt, driver_prompt,
driver_voice_extension_prompt, analysis_check_prompt.
|
| get_reps_baselineA | Serve REPS agent-evaluation baseline material (scenario banks, drivers, checks, eval configs). REPS is Okareo's agent-evaluation workbench: per-pillar baseline
material for evaluating AI agents across R-reasoning, E-execution,
P-performance, and S-security, plus shared explore/ probes and a
profile/ example. The material is published as tagged releases of
the okareo-tools repo; this tool serves the latest release so reps
skills need no local copy of the tree.
Two modes:
- Discovery (omit `path`): list what files exist in the served
release — the full tree, or one area via `pillar`. File lists
change between releases, so always discover before fetching.
- Fetch (provide `path`): return one file's exact content as
published in the release. Use paths verbatim from discovery,
e.g. 'S-security/scenarios/verification-gate.jsonl'.
Every response carries the release tag it was served from (e.g.
'v0.5.1') — record it in evaluation reports as baseline
provenance. `stale: true` means the last release check failed and
the content may lag the newest release.
Args:
pillar: Optional discovery filter. One of: R-reasoning,
E-execution, P-performance, S-security, explore, profile.
Omit to list the entire baseline tree (which also includes
shared material outside these areas).
path: Optional file path (relative to the baseline tree, as
returned by discovery). Provide to fetch that file's
content; omit for discovery.
version: Optional release tag. Currently only the served tag
is available; any other value returns an error naming what
IS available. Omit to accept the served release.
|
| ingest_conversationsA | Submit completed voice conversations to Okareo for monitoring. Each conversation's turns become evaluable data points and any
configured monitors run their checks automatically. Use this to feed
production voice traffic (Retell, Twilio, VAPI, ElevenLabs, or a custom
source) into Okareo monitoring.
Conversations are validated individually: valid ones are ingested and
invalid ones are returned in a "rejected" list — the batch is not
all-or-nothing.
Args:
conversations: List of conversation objects. Each MUST include a
"call_id" and at least one of: "transcript" (a list of
{role, content, timestamp_ms} turns), "audio"
({"type": "url"|"voice_file_id"|"inline_b64", ...}),
"recording_url", or "recording_bytes_b64". Optional per
conversation: "context_token", "metadata", "tags" (tags drive
monitor/filter-group matching), "diarization", "first_turn".
When both a transcript and audio are supplied, the transcript
takes precedence.
project_id: Okareo project ID. Defaults to the account's project.
mut_id: Optional model-under-test ID. Omit for pure monitoring —
data points are then matched to monitors by their tags only.
|
| connect_voice_integrationA | Connect a voice provider so its traffic flows into Okareo monitoring. Creates a provider integration. The returned integration carries an id
and a public_id — pass the provider + public_id to get_voice_webhook_url
to obtain the inbound webhook endpoint to paste into the provider's
console.
Args:
provider: Voice platform — one of: retell, twilio, vapi, elevenlabs.
webhook_auth_type: Webhook authentication type expected by Okareo
for this provider (provider-specific — see Okareo docs).
secrets: Provider-specific secret values (opaque pass-through; the
response never echoes raw secrets, only a summary).
metadata: Optional free-form metadata object.
|
| list_voice_integrationsA | List the voice provider integrations in your Okareo project. Args:
limit: Maximum number of integrations to return (default 20). Use 0
for no limit.
|
| get_voice_integrationA | Retrieve a voice provider integration by id, including its status. Args:
integration_id: The integration's id (from list_voice_integrations).
|
| update_voice_integrationA | Update a voice provider integration's metadata. Args:
integration_id: The integration's id.
metadata: The new metadata object.
|
| rotate_voice_integration_secretA | Rotate a voice provider integration's secrets. Args:
integration_id: The integration's id.
secrets: The new provider secret values. The response returns only
a secret summary, never raw secret values.
|
| delete_voice_integrationB | Delete a voice provider integration by id. Args:
integration_id: The integration's id.
|
| get_voice_webhook_urlA | Get the inbound webhook endpoint for a voice provider. Paste the returned URL into the provider's console so its call traffic
reaches Okareo monitoring.
Args:
provider: Voice platform — one of: retell, twilio, vapi, elevenlabs.
public_id: The integration's public_id (from connect_voice_integration
or get_voice_integration). Required for retell and twilio.
|
| query_analyticsA | Query Okareo's product analytics to understand evaluation trends. Answers questions like "how is my evaluation quality trending" by
aggregating measures across dimensions over a time window.
Args:
measures: Metrics to aggregate. Required. For the ``check_trend``
cube: avg_check_value, issue_rate, error_rate, datapoint_count,
issue_count, error_count, test_run_count, avg_latency, sum_cost,
input_token_count, output_token_count.
dimensions: Optional group-by fields (e.g. ["check.name"],
["target.name"], ["provider"]).
cube: Optional analytics cube name (defaults to ``check_trend``,
currently the only cube).
filters: Optional list of filter objects
``{"member": ..., "operator": ..., "values": [...]}``.
time_range: Optional look-back window — one of LAST_HOUR,
LAST_24_HOURS, LAST_7_DAYS, LAST_14_DAYS, LAST_30_DAYS,
LAST_90_DAYS. If neither time_range nor time_dimensions is
given, defaults to LAST_30_DAYS (the analytics API requires a
time window).
time_dimensions: Optional time bucketing — a list with at most one
entry, e.g. [{"dimension": "test_run.start_time",
"granularity": "day"}] (granularity: hour, day, or week).
include_metadata: When true, also return the available cubes,
dimensions, and measures so the query can be refined.
|
| list_dashboardsA | List the analytics dashboards in your Okareo project. Args:
limit: Maximum number of dashboards to return (default 20). Use 0
for no limit.
|
| get_dashboardA | Retrieve a dashboard's full configuration by name. Args:
name: Name of the dashboard to retrieve.
|
| save_dashboardA | Create or update an analytics dashboard by name (upsert). If a dashboard with this name already exists it is updated; otherwise a
new one is created.
Size each panel with a named ``size`` from the catalog below (PREFERRED
— guarantees a legible layout) and omit positions entirely: panels are
auto-placed in the order given (left-to-right, top-to-bottom on a
12-column grid, wrapping rows, never overlapping).
Size catalog and when to use each:
- ``small-square`` (3x6): single ``stat`` metrics.
- ``half-rectangle`` (6x6): ``line``/``bar``/``area`` trends, two per row.
- ``half-square`` (6x9): ``radar``, ``composed``, denser charts.
- ``full-rectangle`` (12x9): wide time-series comparisons.
- ``full-square`` (12x12): ``table`` panels.
Args:
name: Dashboard name — the upsert key.
panels: Optional list of panel definitions. Each panel is an object:
- ``title`` (str, required): panel heading.
- ``chart_type`` (str, required): one of ``line``, ``bar``,
``composed``, ``area``, ``radar``, ``stat``, ``table``.
- ``query`` (object, required): what to chart —
``{"cube": "check_trend", "measures": [...],
"dimensions": [...], "filters": [...],
"time_dimensions": [...], "order": {...}}``. ``measures`` is
required; everything else is optional. ``cube`` defaults to
``check_trend``. The dashboard ``time_range`` applies to all
panels — panels do NOT carry their own time range.
- ``size`` (str): a catalog name (see above). Required unless
``layout`` is given; wins over ``layout`` w/h if both appear.
- ``layout`` (object): raw grid placement
``{"x": >=0, "y": >=0, "w": >=1, "h": >=1}`` (integers).
Only needed when not using ``size``, or to pin an explicit
position (give both ``x`` and ``y``; with ``size``, w/h are
ignored). Heights below the legibility floor are sized up on
save: ``h >= 6`` when ``w <= 6``, ``h >= 9`` when ``w > 6``.
- ``table_config`` (object, optional): ONLY for
``chart_type == "table"``.
``check_trend`` measures: ``avg_check_value``, ``issue_rate``,
``error_rate``, ``datapoint_count``, ``issue_count``,
``error_count``, ``test_run_count``, ``avg_latency``,
``sum_cost``, ``input_token_count``, ``output_token_count``.
``check_trend`` dimensions: ``check.name``, ``check.id``,
``target.name``, ``target.id``, ``scenario.name``,
``scenario.id``, ``test_run.id``, ``test_run.type``,
``test_run.is_latest_for_target``, ``source``, ``provider``,
``request_model_name``, ``response_model_name``, ``tag``.
Use ``query_analytics(include_metadata=True)`` for the
authoritative, current set.
Example panel::
{"title": "Avg Check Value by Check", "chart_type": "bar",
"query": {"measures": ["avg_check_value"],
"dimensions": ["check.name"]},
"size": "half-rectangle"}
description: Optional dashboard description.
time_range: Optional default look-back window for the whole
dashboard. One of: LAST_HOUR, LAST_24_HOURS, LAST_7_DAYS,
LAST_14_DAYS, LAST_30_DAYS, LAST_90_DAYS. Defaults to
LAST_90_DAYS when omitted.
Returns:
JSON with the saved dashboard and ``action`` (created/updated).
When sizing or dimensions were changed on save (size overriding
layout w/h, or a height floored), an ``adjustments`` list reports
each change: ``{"panel", "field", "from", "to", "reason"}``.
|
| reorder_dashboardsB | Set the display order of dashboards. Args:
ordered_names: Dashboard names in the desired order.
|
| delete_dashboardB | Delete a dashboard by name. Args:
name: Name of the dashboard to delete.
|
| list_tenantsA | List every Okareo organization you have access to in this MCP session. The currently-active organization is marked ``is_current: true``. The
active organization is determined at sign-in (the token this session
presents is already scoped to it). Only available on OAuth-authenticated
sessions; on Bearer-API-key sessions returns
``tenant_selection_requires_oauth``.
|
| switch_tenantB | Change which Okareo organization your session operates against. Organization selection now happens **during sign-in** (feature 030):
when you connect the Okareo MCP you choose which organization to
authorize, and the credential this session uses is already scoped to
it. This tool therefore no longer changes the active organization — to
switch, reconnect/re-authenticate the Okareo MCP from your copilot and
select a different organization when prompted. Use ``list_tenants`` to
see which organization is currently active.
|
