Skip to main content
Glama
Okareo MCP Server

Okareo MCP Server

Official
by okareo-ai

Okareo MCP Server

The Okareo MCP server exposes Okareo's evaluation capabilities as MCP tools, allowing AI coding assistants to create and manage scenarios, register models, run evaluations, and execute multi-turn simulations directly from your editor.

For detailed documentation, see the Okareo MCP docs.

Prerequisites

  • An Okareo account at app.okareo.com

  • A copilot that supports MCP servers (Claude Code, Cursor, VS Code, or GitHub Copilot), or Slackbot in Slack

  • Python 3.10–3.12 (only for the local install modes; not needed for remote)


Related MCP server: Patronus MCP Server

The fastest way to get started. No local install, no Python, no API key in .mcp.json. Browser sign-in handles auth on first connect.

Add to your copilot's MCP config (typically .mcp.json):

{
  "mcpServers": {
    "okareo": {
      "type": "http",
      "url": "https://tools.okareo.com/mcp"
    }
  }
}

Reload the copilot. It will open a browser to Okareo sign-in once; thereafter the copilot stores the token itself. Your .mcp.json contains no secrets.

Fallback (Bearer header — older clients or headless / CI)

For clients that haven't shipped the MCP OAuth flow yet, paste your API key as a bearer header. Prefer the env-var form over an inline literal:

{
  "mcpServers": {
    "okareo": {
      "type": "http",
      "url": "https://tools.okareo.com/mcp",
      "headers": {
        "Authorization": "Bearer ${env:OKAREO_API_KEY}"
      }
    }
  }
}

GitHub Copilot (in VS Code and the Copilot CLI), Slackbot and every other client are set up at docs.okareo.com/mcp/configuration.

Working across multiple Okareo organizations

If you have access to more than one Okareo account, simply Clear Authentication and then re-authenticate with the account you want to work in. You can always ask which account your are in and have acccess to.


Local install (alternative)

Run the MCP server on your own machine. Useful for offline / airgapped environments and for development.

Prefer the Remote MCP section above unless you have a specific reason to install locally (airgapped, custom build, development on this repo). The remote endpoint requires no install and stays current automatically.

For multi-org users, the remote endpoint also exposes tenant management tools (list_tenants, switch_tenant).

Step 1: Set Your API Key

export OKAREO_API_KEY="your-api-key"

Add this to your ~/.zshrc or ~/.bash_profile for persistence.

Step 2: Configure Your Copilot

Claude Code

Add to .mcp.json:

{
  "mcpServers": {
    "okareo": {
      "command": "uvx",
      "args": ["okareo-mcp"],
      "env": {
        "OKAREO_API_KEY": "${OKAREO_API_KEY}"
      }
    }
  }
}

No pre-install needed — uvx handles it automatically.

Cursor

Add to .cursor/mcp.json:

{
  "mcpServers": {
    "okareo": {
      "command": "uvx",
      "args": ["okareo-mcp"]
    }
  }
}

Cursor inherits OKAREO_API_KEY from your shell environment.

Alternative: pip install

If you don't have uv installed:

pip install okareo-mcp

Then use "command": "okareo-mcp" instead of "command": "uvx" with "args": ["okareo-mcp"].


Configuration Reference

Variable

Default

Description

OKAREO_API_KEY

(required)

Your Okareo API key

OKAREO_BASE_URL

https://api.okareo.com

Override for on-prem Okareo backend

TRANSPORT

stdio

Transport: stdio (local) or sse (Docker)

PORT

8000

Port for SSE transport


Available Tools

Scenarios

Tool

Description

save_scenario

Save a named scenario from rows of input/result data (idempotent)

list_scenarios

List all scenarios in the project with names, IDs, and row counts

get_scenario

Retrieve a scenario's metadata and all data rows by name or ID

create_scenario_version

Create a new version of an existing scenario with updated data

preview_delete_scenario

Preview what will be deleted before removing a scenario

delete_scenario

Permanently delete a scenario and all related test data

Generation Models

Tool

Description

list_available_llms

Browse available LLMs from the Okareo registry

register_generation_model

Register a generation model for testing by selecting an LLM from the registry

list_generation_models

List all registered generation models in the project

get_generation_model

Read detailed information about a registered generation model

update_generation_model

Change the LLM a registered generation model points to

delete_generation_model

Remove a registered generation model and all its related test data

Tests & Checks

Tool

Description

list_checks

List available quality checks (built-in and custom) for evaluating model outputs

run_test

Run a quality test that evaluates a model against a scenario using specified checks

list_test_runs

Find past test runs, most recent first. detail_level="summary" (default) or "detailed" for aggregate metrics

get_test_run_results

Load one page of a run's results by ID or name. detail_level="summary" (default) / "detailed" (per-check outcomes + explanations) / "full" (transcripts). Defaults to 20 conversations. Carries a rerun block — the call and config to re-run this simulation

get_conversation_transcript

Retrieve the full conversation transcript for a single data point

reevaluate_test_run

Re-score a completed test run against a (possibly different) set of checks

create_or_update_check

Create or update a quality check by name — model-based, code-based, or audio (upsert)

generate_check

Generate a check from a natural-language description, then save it

get_check

Retrieve a check's full configuration, including its prompt template or code

delete_check

Permanently delete a check by name

calibrate_check

Dry-run a draft check against a finished test run — per-row verdicts and the arguments the check received, saving nothing

Simulations (Multi-Turn)

Tool

Description

create_or_update_target

Create or update a Target — generation model, custom endpoint, voice (OpenAI, Deepgram, Twilio), or vision (a web page in a browser Okareo hosts, with its secrets)

get_target

Retrieve a Target's configuration by name (all types)

list_targets

Find simulation targets (voice, custom_endpoint and hosted vision). Bounded to 20 by default; name_contains filters

delete_target

Remove a simulation target and all its related test data

create_or_update_driver

Define a simulated user persona that will interact with your target

get_driver

Retrieve a Driver's full configuration including the persona prompt

list_drivers

Find Driver personas. Bounded to 20 by default; name_contains filters

list_driver_voices

Discover the voices, voice profiles, and languages available for voice drivers

run_simulation

Run a multi-turn conversation evaluation. With based_on_run_id it inherits the original run's full configuration; any argument you pass overrides that one field

list_simulations

List past simulation runs with optional filters (target, scenario, limit)

Voice Monitoring

Tool

Description

ingest_conversations

Submit completed voice conversations to Okareo for monitoring

connect_voice_integration

Connect a voice provider so its traffic flows into Okareo monitoring

list_voice_integrations

List the voice provider integrations in your project

get_voice_integration

Retrieve a voice provider integration by ID, including its status

update_voice_integration

Update a voice provider integration's metadata

rotate_voice_integration_secret

Rotate a voice provider integration's secrets

delete_voice_integration

Delete a voice provider integration by ID

get_voice_webhook_url

Get the inbound webhook endpoint for a voice provider

Live Traffic

Tool

Description

find_datapoints

Find live-traffic datapoints, newest first, within a window (default 30 days). issues_only, a monitor segment, checks, field filters. detail_level="summary" (default) / "detailed" (check values + explanations) / "full" (whole inputs and results; page cap 50, for pulling traffic to a file); every response stays under 100 KB

get_datapoint

Open one datapoint in full — input, result, tool calls, checks with explanations, metadata — and whether a trace exists

get_trace

The span tree behind a datapoint (or a trace id / context token): names, kinds, depth and timing by default; attributes and metadata with detail_level="full"

Projects

Tool

Description

list_projects

List every project you can work in — id, name, tags, archive state — with the active one marked

select_project

Choose the project to work in for this conversation

create_project

Create a new project

update_project

Rename a project, or replace its tags

archive_project

Hide a project from the project picker. Reversible; nothing is deleted

unarchive_project

Restore an archived project to the picker

clone_project

Copy an existing project's scenarios into a new project (dry-run first)

Okareo has no project delete — in the app, the API, or here. Archiving is the removal, and it only hides the project from the picker: its scenarios, runs, and dashboards stay intact and every tool still works against it.

Analytics & Dashboards

Tool

Description

query_analytics

Query Okareo's product analytics to understand evaluation trends

list_dashboards

List the analytics dashboards in your project

get_dashboard

Retrieve a dashboard's full configuration by name

save_dashboard

Create or update an analytics dashboard by name (upsert)

reorder_dashboards

Set the display order of dashboards

delete_dashboard

Delete a dashboard by name

Tenant Management (remote MCP only)

Tool

Description

list_tenants

List every Okareo organization you have access to in this MCP session

switch_tenant

Doesn't change the active organization; returns how to reconnect and pick another one at sign-in

Documentation & Templates

Tool

Description

get_docs

Query the Okareo documentation system for conceptual or user-legible explanations

get_templates

Retrieve prompt templates for common Okareo patterns (works offline)

get_reps_baseline

Serve REPS agent-evaluation baseline material (scenario banks, drivers, checks, eval configs) from the latest tagged okareo-tools release


Troubleshooting

Symptom

Cause

Fix

okareo-mcp: command not found

Not installed or not in PATH

Run pip install -e . (dev) or use uvx okareo-mcp (user)

Server exits with API key error

OKAREO_API_KEY not set

Export it: export OKAREO_API_KEY="..."

pip install fails on Python 3.13+

Okareo SDK requires Python <3.13

Use Python 3.10–3.12

Copilot can't connect (Docker)

Wrong URL

Ensure URL ends with /sse and port matches

Cursor doesn't pick up API key

Cursor launched from Dock, not terminal

Launch Cursor from terminal: cursor .


Contributing

This repository is a curated public mirror; the canonical source is maintained by Okareo. We welcome issues and consider community pull requests — see CONTRIBUTING.md for how proposed changes are reviewed and ported.

License & Trademarks

The Okareo MCP server source code is licensed under the Apache License 2.0 (see also NOTICE).

"Okareo", the Okareo logo, and Okareo product names are trademarks of Okareo, Inc. and are not covered by the Apache 2.0 license. See TRADEMARK.md for permitted use.

Available Tools

51 tools
connect_voice_integrationConnect Voice IntegrationAInspect

Connect a voice provider so its traffic flows into Okareo monitoring.

    Creates a provider integration. The returned integration carries an id
    and a public_id — pass the provider + public_id to get_voice_webhook_url
    to obtain the inbound webhook endpoint to paste into the provider's
    console.

    Args:
        provider: Voice platform — one of: retell, twilio, vapi, elevenlabs.
        webhook_auth_type: Webhook authentication type expected by Okareo
            for this provider (provider-specific — see Okareo docs).
        secrets: Provider-specific secret values (opaque pass-through; the
            response never echoes raw secrets, only a summary).
        metadata: Optional free-form metadata object.
    

Inputs: secrets (required), provider (required), webhook_auth_type (required), metadata. Returns: result

create_or_update_checkCreate or Update CheckA
Idempotent
Inspect

Create or update a quality check by name (upsert). Supports model-based, code-based, and audio checks.

    Saving to an existing name creates a new version of that check (see
    get_check's "available_versions"). Before writing a prompt_template or
    code_contents from scratch, fetch a worked example with get_templates:
    "boolean_check_prompt", "score_check_prompt", "analysis_check_prompt",
    or "check_code".

    Args:
        name: Unique name for the check.
        description: What the check evaluates.
        check_type: "model" (an LLM judge driven by prompt_template) or
            "code" (a deterministic Python class in code_contents).
        output_type: "pass_fail" (boolean verdict), "score" (numeric, e.g.
            a 1-5 rubric), or "analysis" (free-form qualitative feedback;
            only valid with check_type="model"). For check_type="code" the
            server infers pass_fail vs score from the value evaluate()
            returns (bool vs int/float) — output_type is used only to
            validate the request, not sent to the server. Note: list_checks
            and get_check report this as output_data_type in the server
            vocabulary, where "bool" means pass_fail and "int" means score.
        prompt_template: Required when check_type="model". The judge
            prompt. Inject the runtime data the judge needs with these
            placeholders:
            - {model_output}: the model output being evaluated. In a
              multi-turn conversation this is ONLY the final assistant
              message, not the full conversation.
            - {scenario_input}: the scenario input / source text.
            - {scenario_result}: the reference/expected output.
            - {model_input}: what was sent to the model (prompt or
              messages).
            - {message_history}: the full multi-turn conversation — the
              model_input messages plus the assistant's model_output. Use
              this when the check must judge the whole conversation.
            - {tool_calls}: the tool/function calls the model just made.
            - {tools}: the tool definitions/schema available to the model.
            - {model_output_metadata}: metadata attached to the most
              recent model output.
            - {simulation_message_history}: full conversation history
              reconstructed from trace metadata. Only populated for traced
              (ingested) conversations; for simulations and evaluations
              use {message_history}.
            The legacy {generation} placeholder is deprecated — use
            {model_output} instead.
        code_contents: Required when check_type="code" (output_type
            "pass_fail" or "score" only). Python source defining
            `class Check(CodeBasedCheck)` with a
            `@staticmethod def evaluate(...) -> CheckResponse` method.
            Start from `from okareo.checks import CodeBasedCheck,
            CheckResponse`. evaluate() may declare any subset of these
            parameters: model_output, scenario_input, scenario_result,
            metadata, model_input. Return CheckResponse(score=...,
            explanation=...) where score is a bool for pass_fail or an
            int/float for score. See get_templates("check_code") for
            complete examples.
        is_audio: Set to true for audio/voice evaluation. Only valid with
            check_type="model".
        tags: Optional list of string tags to organize the check. Tags are
            stored with the check and returned by get_check.
    

Inputs: name (required), check_type (required), description (required), output_type (required), +4 more. Returns: result

create_or_update_driverCreate or Update DriverA
Idempotent
Inspect

Define a simulated user persona that will interact with your target.

    Creates or updates a Driver by name (upsert). Author ONLY the core
    persona sections in prompt_template, in this order:

    - `## Persona` — who the simulated user is (static character).
    - `## Scenario Details` — contains the scenario reference
      (`{scenario_input}` or a specific path like
      `{scenario_input.objectives}`), placed immediately before
      Objectives. This is how each scenario row's data reaches the
      conversation.
    - `## Objectives` — WHAT the driver is trying to accomplish, written
      from the driver's goal (not from scenario variables).
    - `## Soft Tactics` — HOW the driver probes, escalates, and stops.

    Do NOT author Hard Rules, a Turn-End Checklist, or Conversation
    Behavior sections: the MCP automatically appends the platform's
    canonical versions of those blocks (including the language rule
    matching `language`) — the same blocks the Okareo UI appends to
    generated drivers. Any caller-authored variant of these sections is
    replaced by the canonical text, and repeated updates never duplicate
    the blocks.

    For voice agents, configure how the simulated user speaks with `voice`,
    `voice_profile`, `voice_instructions`, and `language`. Call
    list_driver_voices first to discover valid voice and profile values.

    Args:
        name: Unique name for this driver.
        prompt_template: The core persona prompt (Persona, Scenario
            Details, Objectives, Soft Tactics — see above). Hard Rules and
            Conversation Behavior are appended automatically.
        model_id: Foundation model to power the driver (defaults to project default).
        temperature: Response randomness, default 0.6.
        voice_instructions: Free-text speaking instructions for voice simulations
            (tone, pace, accent). Not validated against the voice catalog.
        voice_profile: Voice profile name for voice simulations. Validated
            against the catalog from list_driver_voices.
        voice: Voice identifier for voice simulations. Validated against the
            catalog from list_driver_voices.
        language: Language the driver responds in, as the bare ISO code
            the voice catalog serves (e.g. "en", "es", "ja"); regional
            variants like "fr-CA" are accepted when their base code
            matches the voice's language. When a `voice` is set and
            language is omitted, it is derived from that voice's catalog
            language (disclosed as `language_derived_from_voice` in the
            response); a value conflicting with the voice's language is
            rejected. Also drives the appended Hard Rules language rule.
    

Inputs: name (required), prompt_template (required), voice, language, +4 more. Returns: result

create_or_update_targetCreate or Update TargetA
Idempotent
Inspect

Create or update a Target — the AI system you want to evaluate in a simulation.

    Calling create_or_update_target with the same name as an existing Target will
    **fully replace** its configuration — caller must re-specify all desired fields.
    Supported types: 'generation' (foundation model), 'custom_endpoint' (your own
    REST API), and 'voice' (voice-based targets reached by phone via Twilio or
    over SIP).

    **Cloning workflow**: this tool accepts the same key structure that `get_target`
    returns, so you can read an existing Target, change `name`, swap in real values
    for any field whose value is `"***REDACTED***"`, and pass the result here as
    kwargs. Calls that still contain the redaction sentinel are rejected with an
    error naming each offending path; the sentinel is never forwarded to the backend.

    Args:
        name: Unique name for this target.
        type: Target type — 'generation', 'custom_endpoint', or 'voice'.

        model_id: (generation targets) Foundation model ID, e.g. 'gpt-4o-mini'.
        temperature: (generation targets) Response randomness, default 0.
        system_prompt_template: (generation targets) System instructions; mustache
            syntax supported, e.g. '{scenario_input}'.
        user_prompt_template: (generation targets) User prompt template.
        dialog_template: (generation targets) Dialog formatting template.
        tools: (generation targets) Tool definitions for function calling.

        next_message_params: (custom_endpoint) Nested HTTP config for each
            conversation turn. Required keys: 'url', 'method'. Optional:
            'headers', 'body', 'status_code', 'response_message_path',
            'response_session_id_path', 'response_tool_calls_path'.
            All response path values MUST use dot-path notation starting with
            'response.' — e.g., 'response.message', 'response.choices[0].message.content',
            'response.choices[0].tool_calls'. Never use bare property names.
            For SSE/streaming endpoints, include a 'streaming' object with:
              - 'stop': array of stop conditions (OR semantics — any match ends
                the stream). Each has 'value' (required) and optional 'path'
                (dot-path into JSON chunk). Without 'path', matches raw SSE data.
              - 'select': array of select conditions (AND semantics — all must
                match for a chunk's content to be extracted). Each requires
                'path' and 'value'.
            When streaming, set response_message_path to the chunk field
            (e.g., 'response.choices[0].delta.content').
        start_session_params: (custom_endpoint, optional) Nested HTTP config to
            initialise a session. Required key: 'url'. Optional: 'method',
            'headers', 'body', 'status_code', 'response_session_id_path'
            (dot-path starting with 'response.', e.g. 'response.id'),
            'response_message_path'. Supports 'streaming' object (same
            structure as next_message_params.streaming).
        end_session_params: (custom_endpoint, optional) Nested HTTP config to
            close a session after the last turn.
        auth_params: (custom_endpoint, optional) Token-based authorization config.
            Required keys when provided: 'url', 'method', 'response_access_token_path'
            (dot-path starting with 'response.', e.g. 'response.access_token').
            Optional: 'headers', 'body', 'status_code'.
        sensitive_fields: (custom_endpoint, optional) List of dot-path strings for
            secret fields within auth_params (e.g., 'auth_params.body.client_id').
            The MCP auto-generates entries for top-level auth_params keys; use this
            for deeper paths. To remove auth from an existing target, call
            create_or_update_target again without auth_params.
        max_parallel_requests: (custom_endpoint, twilio) Concurrency limit. This is
            the same setting the Okareo web UI labels "max concurrency".

        edge_type: (voice targets) How Okareo reaches the voice agent —
            'twilio' (dial a phone number) or 'sip' (call a SIP URI).

        to_phone_number: (voice twilio) Destination phone number (required).
        account_sid: (voice twilio, custom only) Twilio account SID. If provided,
            auth_token and from_phone_number are also required (all-or-nothing).
            Omit for generic Twilio targets using Okareo's managed integration.
        auth_token: (voice twilio, custom only) Twilio auth token. Required with
            account_sid and from_phone_number.
        from_phone_number: (voice twilio, custom only) Caller phone number. Required
            with account_sid and auth_token.

        sip_uri: (voice sip) Destination SIP URI (required), e.g.
            "sip:agent@your-domain.example.com". Use this to test any voice
            agent reachable over SIP — for example one fronted by Daily,
            Vapi, LiveKit, or a SIP trunk.
        sip_username: (voice sip, optional) SIP authentication username.
        sip_password: (voice sip, optional) SIP authentication password
            (stored as a sensitive field).
    

Inputs: name (required), type (required), tools, sip_uri, +18 more. Returns: result

create_scenario_versionCreate Scenario VersionAInspect

Create a new version of an existing scenario with updated data.

    Automatically determines the next version number (e.g., 'my-test-v2',
    'my-test-v3'). The original scenario is treated as version 1.

    Args:
        base_name: Name of the original scenario to create a version of.
        rows: List of data rows for the new version, each with 'input' and 'result'.
    

Inputs: rows (required), base_name (required). Returns: result

delete_checkDelete CheckA
Destructive
Inspect

Permanently delete a check by name.

    Args:
        name: Name of the check to delete.
    

Inputs: name (required). Returns: result

delete_dashboardDelete DashboardB
Destructive
Inspect

Delete a dashboard by name.

    Args:
        name: Name of the dashboard to delete.
    

Inputs: name (required). Returns: result

delete_generation_modelDelete Generation ModelA
Destructive
Inspect

Remove a registered generation model and all its related test data.

    Permanently deletes the generation model and cascades to associated
    test runs and test data points. This cannot be undone.

    Args:
        name: Name of the registered generation model to delete.
    

Inputs: name (required). Returns: result

delete_scenarioDelete ScenarioA
Destructive
Inspect

Permanently delete a scenario and all related test data.

    Both scenario_id and name are required. Use preview_delete_scenario first
    to see what will be deleted before confirming.

    Args:
        scenario_id: The ID of the scenario to delete (from preview_delete_scenario).
        name: The name of the scenario to delete.
    

Inputs: name (required), scenario_id (required). Returns: result

delete_targetDelete TargetA
Destructive
Inspect

Remove a simulation target and all its related test data.

    Permanently deletes the target and cascades to associated
    test runs and test data points. This cannot be undone.

    Args:
        name: Name of the target to delete.
    

Inputs: name (required). Returns: result

delete_voice_integrationDelete Voice IntegrationB
Destructive
Inspect

Delete a voice provider integration by id.

    Args:
        integration_id: The integration's id.
    

Inputs: integration_id (required). Returns: result

generate_checkGenerate CheckAInspect

Generate a check from a natural language description. Uses AI to create the prompt template (model checks) or Python code (code checks), then saves the check.

    Use this when you only have a description of what to evaluate. When
    you already know the exact prompt template or Python code the check
    should use, call create_or_update_check directly instead. The
    generated prompt/code is returned in the response — review it and
    refine with create_or_update_check if needed.

    Args:
        name: Name for the generated check.
        description: Natural language description of what to evaluate
            (e.g., "check if the response is toxic"). The more specific
            the description, the better the generated check.
        output_type: "pass_fail" (boolean verdict), "score" (numeric), or
            "analysis" (free-form qualitative feedback; model checks only).
        check_type: "model" (LLM judge) or "code" (deterministic Python).
        requires_scenario_input: Set true when the evaluation must compare
            the output against the scenario input. The generated check
            will reference {scenario_input} and only works on runs whose
            scenarios provide it.
        requires_scenario_result: Set true when the evaluation must
            compare the output against the expected result. The generated
            check will reference {scenario_result} and only works on runs
            whose scenarios provide it.
    

Inputs: name (required), description (required), check_type, output_type, +2 more. Returns: result

get_checkGet CheckA
Read-onlyIdempotent
Inspect

Retrieve the full configuration of a check by name, including its prompt template or code contents.

    Args:
        name: Name of the check to retrieve.
        version: Optional check version number to pin. Omit (or leave null)
            for the most recent version. The response always lists every
            available version under "available_versions".
    

Inputs: name (required), version. Returns: result

get_conversation_transcriptGet Conversation TranscriptA
Read-onlyIdempotent
Inspect

Retrieve the full conversation transcript for a single data point.

    Use this after get_test_run_results to drill into a specific
    conversation. Provide either scenario_index (1-based, from the
    scores summary) or test_id (UUID) to identify the conversation.

    Returns the complete message transcript (model_input), final
    output (model_result), per-turn check scores (metric_value),
    and the scenario seed data.

    Args:
        test_run_id: The UUID of the test run.
        scenario_index: 1-based index of the conversation within
            the test run. Visible in get_test_run_results output.
        test_id: UUID of the specific data point. Alternative to
            scenario_index.
    

Inputs: test_run_id (required), test_id, scenario_index. Returns: result

get_dashboardGet DashboardA
Read-onlyIdempotent
Inspect

Retrieve a dashboard's full configuration by name.

    Args:
        name: Name of the dashboard to retrieve.
    

Inputs: name (required). Returns: result

get_docsQuery Okareo DocumentationA
Read-onlyIdempotent
Inspect

Query the Okareo documentation system for information about Okareo primitives and workflows.

    Use this tool when the agent or user needs to understand how Okareo
    concepts work — Scenarios, Checks, Targets, Drivers, Evaluations,
    and Simulations.

    Two modes are available:
    - 'conceptual': Detailed technical documentation for agent reasoning.
      Default top_k=5 (returns up to 5 documentation entries).
    - 'user_legible': Plain-language explanations for human users.
      Default top_k=3 (returns up to 3 documentation entries).

    If the Okareo documentation service is unavailable (e.g. air-gapped
    environment), the tool returns a helpful error suggesting get_templates
    as a fallback.

    Args:
        query: The question to ask the Okareo documentation system. Be
            specific — e.g., 'How do Checks and Evaluations work together?'
            or 'What is a Driver persona?'
        mode: Documentation mode — 'conceptual' or 'user_legible'.
        top_k: Number of documentation entries to return. Defaults to 5
            for conceptual mode, 3 for user_legible mode. Maximum 10.
    

Inputs: mode (required), query (required), top_k. Returns: result

get_driverGet DriverA
Read-onlyIdempotent
Inspect

Retrieve a driver persona you've already configured.

    Retrieves a Driver by name, returning its full configuration including the
    persona prompt.

    Args:
        name: Name of the driver to retrieve.
    

Inputs: name (required). Returns: result

get_generation_modelGet Generation ModelA
Read-onlyIdempotent
Inspect

Read detailed information about a registered generation model.

    Returns the generation model's target LLM configuration, tags, creation
    time, and any warnings (e.g., if the target LLM has been deprecated).

    Args:
        name: Name of the registered generation model.
    

Inputs: name (required). Returns: result

get_reps_baselineGet REPS Baseline MaterialA
Read-onlyIdempotent
Inspect

Serve REPS agent-evaluation baseline material (scenario banks, drivers, checks, eval configs).

    REPS is Okareo's agent-evaluation workbench: per-pillar baseline
    material for evaluating AI agents across R-reasoning, E-execution,
    P-performance, and S-security, plus shared explore/ probes and a
    profile/ example. The material is published as tagged releases of
    the okareo-tools repo; this tool serves the latest release so reps
    skills need no local copy of the tree.

    Two modes:
    - Discovery (omit `path`): list what files exist in the served
      release — the full tree, or one area via `pillar`. File lists
      change between releases, so always discover before fetching.
    - Fetch (provide `path`): return one file's exact content as
      published in the release. Use paths verbatim from discovery,
      e.g. 'S-security/scenarios/verification-gate.jsonl'.

    Every response carries the release tag it was served from (e.g.
    'v0.5.1') — record it in evaluation reports as baseline
    provenance. `stale: true` means the last release check failed and
    the content may lag the newest release.

    Args:
        pillar: Optional discovery filter. One of: R-reasoning,
            E-execution, P-performance, S-security, explore, profile.
            Omit to list the entire baseline tree (which also includes
            shared material outside these areas).
        path: Optional file path (relative to the baseline tree, as
            returned by discovery). Provide to fetch that file's
            content; omit for discovery.
        version: Optional release tag. Currently only the served tag
            is available; any other value returns an error naming what
            IS available. Omit to accept the served release.
    

Inputs: path, pillar, version. Returns: result

get_scenarioGet ScenarioA
Read-onlyIdempotent
Inspect

Read a scenario's metadata and all data rows.

    Look up by name or scenario ID. Returns scenario details and all
    input/result data rows.

    Args:
        name: Name of the scenario to retrieve.
        scenario_id: ID of the scenario to retrieve. Takes precedence over name.
    

Inputs: name, scenario_id. Returns: result

get_targetGet TargetA
Read-onlyIdempotent
Inspect

Check the current configuration of a test target.

    Retrieves a Target by name. Works for all target types (Generation,
    Custom Endpoint, and Voice).

    For **custom_endpoint** Targets, the response is a flat envelope whose
    keys mirror the kwargs accepted by `create_or_update_target`, so a
    copilot can read the result, swap in a new name + secrets, and feed it
    back to create to clone the Target. Fields the backend keeps secret
    (those listed in `sensitive_fields`) appear with the literal value
    `"***REDACTED***"` — these MUST be replaced with real values before
    calling `create_or_update_target`, which rejects payloads still
    containing the sentinel.

    The `max_parallel_requests` field on custom_endpoint Targets is the
    same setting the Okareo web UI labels "max concurrency".

    For generation and voice Targets, the response shape is unchanged
    (kept stable for existing callers).

    Args:
        name: Name of the target to retrieve.
    

Inputs: name (required). Returns: result

get_templatesGet Okareo TemplatesA
Read-onlyIdempotent
Inspect

Retrieve prompt templates for common Okareo patterns.

    Returns starter templates for building Okareo test components. These
    templates are served as static content from the MCP — no network calls
    required. Always available, even in air-gapped environments.

    Available templates:
    - basic_scenario: Template for creating a basic Okareo test scenario
    - boolean_check_prompt: Template for a pass/fail (boolean) check prompt
    - score_check_prompt: Template for a scored check prompt
    - check_code: Template for a code-based check (Python function)
    - target_validate_check_prompt: Template for validating target output
    - driver_prompt: Template for a Driver persona prompt
    - driver_voice_extension_prompt: Template for voice interaction extensions
    - analysis_check_prompt: Template for an analysis check (qualitative feedback)

    Args:
        template_name: Template identifier to retrieve. Omit to get a
            lightweight listing of all available templates (names and
            descriptions only). Provide a template_name to get the full
            template content. Valid values: basic_scenario,
            boolean_check_prompt, score_check_prompt, check_code,
            target_validate_check_prompt, driver_prompt,
            driver_voice_extension_prompt, analysis_check_prompt.
    

Inputs: template_name. Returns: result

get_test_run_resultsGet Test Run ResultsA
Read-onlyIdempotent
Inspect

Load the results of a specific test run.

    Look up by test run ID (UUID) or by name (returns the most recent run
    matching that name). Returns aggregate metrics and per-row check scores.

    By default, conversation transcripts (model_input/model_result) are
    excluded to keep responses concise. Set include_transcripts=True to
    include full transcripts. Use get_conversation_transcript to inspect
    a single conversation's transcript without loading all of them.

    Supports pagination via limit and offset for large result sets.

    Args:
        test_run_id: The UUID of the test run. Takes precedence over name.
        name: The name of the test run. Returns the most recent match.
        include_transcripts: Include full model_input and model_result in
            each data point. Defaults to False (scores only). Set True for
            full conversation transcripts.
        limit: Maximum number of data points to return. 0 (default) returns
            all data points. Use with offset for pagination.
        offset: Number of data points to skip. Defaults to 0.
    

Inputs: name, limit, offset, test_run_id, +1 more. Returns: result

get_voice_integrationGet Voice IntegrationA
Read-onlyIdempotent
Inspect

Retrieve a voice provider integration by id, including its status.

    Args:
        integration_id: The integration's id (from list_voice_integrations).
    

Inputs: integration_id (required). Returns: result

get_voice_webhook_urlGet Voice Webhook URLA
Read-onlyIdempotent
Inspect

Get the inbound webhook endpoint for a voice provider.

    Paste the returned URL into the provider's console so its call traffic
    reaches Okareo monitoring.

    Args:
        provider: Voice platform — one of: retell, twilio, vapi, elevenlabs.
        public_id: The integration's public_id (from connect_voice_integration
            or get_voice_integration). Required for retell and twilio.
    

Inputs: provider (required), public_id. Returns: result

ingest_conversationsIngest ConversationsAInspect

Submit completed voice conversations to Okareo for monitoring.

    Each conversation's turns become evaluable data points and any
    configured monitors run their checks automatically. Use this to feed
    production voice traffic (Retell, Twilio, VAPI, ElevenLabs, or a custom
    source) into Okareo monitoring.

    Conversations are validated individually: valid ones are ingested and
    invalid ones are returned in a "rejected" list — the batch is not
    all-or-nothing.

    Args:
        conversations: List of conversation objects. Each MUST include a
            "call_id" and at least one of: "transcript" (a list of
            {role, content, timestamp_ms} turns), "audio"
            ({"type": "url"|"voice_file_id"|"inline_b64", ...}),
            "recording_url", or "recording_bytes_b64". Optional per
            conversation: "context_token", "metadata", "tags" (tags drive
            monitor/filter-group matching), "diarization", "first_turn".
            When both a transcript and audio are supplied, the transcript
            takes precedence.
        project_id: Okareo project ID. Defaults to the account's project.
        mut_id: Optional model-under-test ID. Omit for pure monitoring —
            data points are then matched to monitors by their tags only.
    

Inputs: conversations (required), mut_id, project_id. Returns: result

list_available_llmsList Available LLMsA
Read-onlyIdempotent
Inspect

Browse available LLMs from the Okareo registry.

    Returns all LLMs that can be used when registering a generation model
    for testing. Each entry has a name, display name, and provider. Use a
    model_name from this list when calling register_generation_model.
    

Inputs: no named parameters. Returns: result

list_checksList ChecksA
Read-onlyIdempotent
Inspect

List available quality checks, grouped by category.

    Returns checks (both built-in and custom) available in your Okareo
    account, organized into `checks_by_category` using the platform's
    `__category:<Category>` tags; checks with no category appear under
    `uncategorized`. Select checks from the category matching your task
    AND modality: voice-specific categories (e.g. voice/audio quality)
    apply to voice simulations, while checks outside voice-specific
    categories are generally useful for both chat and voice. A check
    carrying multiple categories appears under each of them.

    Each check has a name, description, and output_data_type.
    output_data_type uses the server vocabulary: "bool" is a pass/fail
    check and "int" is a scored check — these correspond to output_type
    "pass_fail" and "score" in create_or_update_check and generate_check.
    Use these check names with run_test to evaluate model quality.

    Args:
        limit: Maximum number of checks to return (default 20), applied to
            the total before grouping. Use 0 for no limit.
        all_versions: When false (default), returns only the latest version of
            each check. When true, returns the full version history of every
            check, each entry annotated with its version number.
    

Inputs: limit, all_versions. Returns: result

list_dashboardsList DashboardsA
Read-onlyIdempotent
Inspect

List the analytics dashboards in your Okareo project.

    Args:
        limit: Maximum number of dashboards to return (default 20). Use 0
            for no limit.
    

Inputs: limit. Returns: result

list_driversList DriversA
Read-onlyIdempotent
Inspect

See what driver personas are available in this project.

Returns all Drivers with their names, IDs, model, and temperature.

Inputs: no named parameters. Returns: result

list_driver_voicesList Driver VoicesA
Read-onlyIdempotent
Inspect

Discover the voices, voice profiles, and languages available for configuring voice-capable drivers.

    Call this before create_or_update_driver when building a voice agent
    simulation, so you can pass valid `voice`, `voice_profile`, and
    `language` values.

    Each entry in `voices` carries selection metadata — use it to pick
    the voice:
    - `language`: bare ISO code (e.g. "en", "es", "ja"). The driver's
      `language` is derived from the selected voice's language when
      omitted, and must not conflict with it.
    - `accent`: free-text accent label present on many voices (e.g.
      "British", "Mexican", "Parisian", "Southern US"). To satisfy an
      accent request, select a voice whose `accent` matches — writing
      accent instructions into `voice_instructions` does NOT change the
      TTS voice.
    - `gender`: e.g. "feminine" / "masculine".

    `voice_profiles` are emotion/delivery presets (happy, angry,
    sarcastic, ...) — they shape affect, not accent or language.
    `languages` lists the distinct voice languages available.
    

Inputs: no named parameters. Returns: result

list_generation_modelsList Generation ModelsA
Read-onlyIdempotent
Inspect

Browse all registered generation models in the project.

Returns generation model names, IDs, target LLM configurations, and creation timestamps. Use this to see what generation models are available for testing.

Inputs: no named parameters. Returns: result

list_scenariosList ScenariosA
Read-onlyIdempotent
Inspect

List scenarios in the project, most recent first.

    Returns scenario names, IDs, tags, row counts, and creation dates.
    Use this to discover existing scenarios before running a test.

    Args:
        limit: Maximum number of scenarios to return (default 20).
            Set to 0 to return all scenarios.
    

Inputs: limit. Returns: result

list_simulationsList SimulationsA
Read-onlyIdempotent
Inspect

List past simulation runs in the project.

    Returns simulation run names, IDs, timestamps, and status, sorted by
    most recent first. Defaults to the 10 most recent runs in summary mode.

    Use detail_level="detailed" to include model_metrics and additional
    fields (limit is capped to 5 in detailed mode to prevent overflow).

    Use get_test_run_results with the returned test_run_id to retrieve
    per-row scores (transcripts excluded by default). Then use
    get_conversation_transcript with a scenario_index to inspect
    individual conversation transcripts.

    Args:
        target_name: Optional filter — only show simulation runs using
            this target.
        scenario_name: Optional filter — only show simulation runs using
            this scenario.
        limit: Maximum number of runs to return, sorted by most recent
            first. Defaults to 10. Set to 0 to return all runs.
        detail_level: "summary" (default) returns compact results without
            model_metrics. "detailed" returns full results with metrics
            (limit capped to 5).
    

Inputs: limit, target_name, detail_level, scenario_name. Returns: result

list_targetsList TargetsA
Read-onlyIdempotent
Inspect

Browse all simulation targets available in this project.

    Returns all simulation targets (voice and custom_endpoint types)
    created via create_or_update_target. Does not include generation models
    registered via register_generation_model — use list_generation_models
    for those.
    

Inputs: no named parameters. Returns: result

list_tenantsList TenantsA
Read-onlyIdempotent
Inspect

List every Okareo organization you have access to in this MCP session.

    The currently-active organization is marked ``is_current: true``. The
    active organization is determined at sign-in (the token this session
    presents is already scoped to it). Only available on OAuth-authenticated
    sessions; on Bearer-API-key sessions returns
    ``tenant_selection_requires_oauth``.
    

Inputs: no named parameters. Returns: result

list_test_runsList Test RunsA
Read-onlyIdempotent
Inspect

List past test runs in the project.

    Returns test run names, IDs, timestamps, status, and summary scores,
    sorted by most recent first. Defaults to the 10 most recent runs.
    Optionally filter by model name, scenario name, or type.

    For simulation runs (type MULTI_TURN), use get_test_run_results with the
    returned test_run_id to retrieve full conversation transcripts and per-turn
    check scores.

    Args:
        model_name: Optional filter — only show test runs using this model.
        scenario_name: Optional filter — only show test runs using this scenario.
        limit: Maximum number of runs to return, sorted by most recent first.
            Defaults to 10. Set to 0 to return all runs.
        simulation_only: When True, return only MULTI_TURN simulation runs.
            Useful for browsing past simulation results without NL_GENERATION or
            other test run types appearing in the list.
    

Inputs: limit, model_name, scenario_name, simulation_only. Returns: result

list_voice_integrationsList Voice IntegrationsA
Read-onlyIdempotent
Inspect

List the voice provider integrations in your Okareo project.

    Args:
        limit: Maximum number of integrations to return (default 20). Use 0
            for no limit.
    

Inputs: limit. Returns: result

preview_delete_scenarioPreview Scenario DeletionA
Read-onlyIdempotent
Inspect

Preview what will be deleted before removing a scenario.

    Shows the scenario details and count of related test runs that will
    also be deleted. Use delete_scenario to confirm deletion after reviewing.

    Args:
        name: Name of the scenario to preview deletion for.
        scenario_id: ID of the scenario. Takes precedence over name.
    

Inputs: name, scenario_id. Returns: result

query_analyticsQuery AnalyticsA
Read-onlyIdempotent
Inspect

Query Okareo's product analytics to understand evaluation trends.

    Answers questions like "how is my evaluation quality trending" by
    aggregating measures across dimensions over a time window.

    Args:
        measures: Metrics to aggregate. Required. For the ``check_trend``
            cube: avg_check_value, issue_rate, error_rate, datapoint_count,
            issue_count, error_count, test_run_count, avg_latency, sum_cost,
            input_token_count, output_token_count.
        dimensions: Optional group-by fields (e.g. ["check.name"],
            ["target.name"], ["provider"]).
        cube: Optional analytics cube name (defaults to ``check_trend``,
            currently the only cube).
        filters: Optional list of filter objects
            ``{"member": ..., "operator": ..., "values": [...]}``.
        time_range: Optional look-back window — one of LAST_HOUR,
            LAST_24_HOURS, LAST_7_DAYS, LAST_14_DAYS, LAST_30_DAYS,
            LAST_90_DAYS. If neither time_range nor time_dimensions is
            given, defaults to LAST_30_DAYS (the analytics API requires a
            time window).
        time_dimensions: Optional time bucketing — a list with at most one
            entry, e.g. [{"dimension": "test_run.start_time",
            "granularity": "day"}] (granularity: hour, day, or week).
        include_metadata: When true, also return the available cubes,
            dimensions, and measures so the query can be refined.
    

Inputs: measures (required), cube, filters, dimensions, +3 more. Returns: result

reevaluate_test_runRe-evaluate Test RunAInspect

Re-score a completed test run against a set of checks.

    Re-runs checks against an already-finished test run without re-executing
    the original model or simulation, and without changing the original
    run's results. Useful after a check definition changed, or to score an
    existing run against additional checks.

    Args:
        test_run_id: UUID or name of a completed test run.
        checks: Optional list of check names (or IDs) to score against.
            When omitted, the run's existing checks are re-run.
    

Inputs: test_run_id (required), checks. Returns: result

register_generation_modelRegister Generation ModelA
Idempotent
Inspect

Register a generation model for testing by selecting an LLM from the registry.

    Creates a generation model (Model Under Test) that points to a specific LLM
    (e.g., 'azure/gpt-4o-mini'). Use list_available_llms to see available
    LLMs. The registered generation model can then be used with run_test.

    Args:
        name: A human-readable name for this generation model (e.g., 'my-chatbot').
        model_name: The LLM from the registry (from list_available_llms).
    

Inputs: name (required), model_name (required). Returns: result

reorder_dashboardsReorder DashboardsB
Idempotent
Inspect

Set the display order of dashboards.

    Args:
        ordered_names: Dashboard names in the desired order.
    

Inputs: ordered_names (required). Returns: result

rotate_voice_integration_secretRotate Voice Integration SecretA
Destructive
Inspect

Rotate a voice provider integration's secrets.

    Args:
        integration_id: The integration's id.
        secrets: The new provider secret values. The response returns only
            a secret summary, never raw secret values.
    

Inputs: secrets (required), integration_id (required). Returns: result

run_simulationRun SimulationAInspect

Run a multi-turn conversation evaluation of your AI agent.

    Combines a Target (the system under test), a Driver (the simulated user),
    and a Scenario (the test cases) to generate realistic multi-turn conversations
    and evaluate them with quality checks.

    Returns promptly so the call never times out on long runs. Short runs that
    finish within the buffer window return ``status: "finished"`` with results
    ready; longer runs return ``status: "running"`` with the ``test_run_id``,
    ``app_link``, and an ``estimated_runtime`` — the run continues to completion
    on its own. In both cases, poll get_test_run_results with the returned
    test_run_id for scores, and get_conversation_transcript for transcripts.

    To rerun a previous simulation — keeping its configuration but changing one or
    more parameters — pass based_on_run_id with the original run's ID and supply
    only the values you want to override. If scenario_name or target_name are
    omitted and based_on_run_id is provided, they will be resolved from the
    original run.

    For custom_endpoint Targets: an exception raised during the run (for
    example the endpoint erroring mid-conversation) FAILS the run — it is
    reported as a failed simulation, not silently skipped.

    **Voice augmentations** — for voice Targets, the `augmentation` parameter
    applies realistic acoustic and conversational effects. Six top-level keys:
    `cap`, `directed_speech`, `secondary_speaker`, `backchannel`, `barge_in`,
    plus the composable `noise`. **Composition rule**: at most one non-noise
    strategy may be active, optionally combined with `noise`. Augmentations
    apply only to voice Targets — calls against generation or custom_endpoint
    Targets with an augmentation block are rejected. Field-level errors
    (out-of-range probability, missing required field, swapped offsets, unknown
    strategy) are returned by the MCP before any backend call.

    Strategy required / optional fields (numeric ranges in brackets):
      - cap: probability [0.0, 1.0] required. pause_ms [0, 10000] optional.
      - directed_speech: probability [0.0, 1.0] required. lpf_cutoff_hz (>0),
        gain_db [-40.0, 0.0], sample_rate (>0), prompt, reverb_preset optional.
      - secondary_speaker: probability [0.0, 1.0] AND secondary_voice (non-empty
        string) required. inter_speaker_pause_ms [0, 5000], lpf_cutoff_hz (>0),
        gain_db [-40.0, 0.0], sample_rate (>0), secondary_prompt,
        secondary_voice_instructions, secondary_reverb_preset optional.
      - backchannel: utterance (non-empty string) required. probability
        [0.0, 1.0], min_offset_ms (>=0), max_offset_ms (>= min_offset_ms),
        seed optional.
      - barge_in: prompt (non-empty string) required. probability [0.0, 1.0],
        min_offset_ms (>=0), max_offset_ms (>= min_offset_ms), seed optional.
      - noise: noise_profile (non-empty string) AND noise_snr_db (number)
        required. seed optional.

    For copy-paste examples and the full reference, call
    `get_templates(["voice_augmentations"])`.

    Args:
        name: Human-readable name for this simulation run.
        scenario_name: Name of the scenario to use. Required unless based_on_run_id
            is provided and the original run's scenario can be resolved.
        target_name: Name of the target to evaluate. Required unless based_on_run_id
            is provided and the original run's target can be resolved.
        driver_name: Name of the driver persona. If omitted, the project default
            driver is used.
        checks: List of check names to apply (from list_checks). Pick from
            the list_checks category matching the task and modality —
            voice-specific categories for voice simulations, categories
            outside them for either modality; never chat-only checks for
            audio (or vice versa). Every simulation runs with at least one
            check: when omitted or empty, the benign code-based "latency"
            performance check is applied automatically and the response
            discloses the substitution via `default_check_applied`.
            Supplied checks are used unchanged.
        repeats: Number of times to run each scenario row, default 1.
        max_turns: Maximum conversation turns per simulation, default 5.
        first_turn: Who speaks first — 'target' or 'driver', default 'target'.
        based_on_run_id: ID of a previous simulation run to reuse parameters from.
            Explicitly supplied values override the original run's parameters.
        augmentation: (voice Targets only) Voice augmentation block. See the
            "Voice augmentations" section above for keys and ranges. An empty
            dict is treated as no augmentation.
        turn_transition_time: Milliseconds of pause between turns. Forwarded to
            the backend as-is; SDK default (1000) is used when omitted.
        silence_timeout_ms: The target reply timeout — how patient Okareo
            is before indicating that the target can't respond. Do NOT set
            or change this value unless the user specifically directs it;
            it should be 10000 ms in nearly all cases. It exists to
            accommodate untuned targets with very long tool calls, during
            which the Driver waits patiently. It does NOT change how fast
            Okareo responds, and lowering it does not speed anything up —
            a slow simulation is not a reason to change it. Forwarded to
            the backend; backend default is used when omitted.
        checks_at_every_turn: When True, checks are evaluated per turn (not
            only at end of run).
        stop_check: Early-stop config: `{"check_name": str, "stop_on": <value>}`.
            The run halts as soon as the named check returns `stop_on`.
    

Inputs: name (required), checks, repeats, max_turns, +10 more. Returns: result

run_testRun TestAInspect

Submit a quality test that evaluates a model against a scenario using checks.

    Returns promptly so the call never times out on long runs. Short runs return
    ``status: "finished"`` with results ready; longer runs return
    ``status: "running"`` with the ``test_run_id`` and ``app_link`` — the run
    continues to completion on its own. In both cases, poll get_test_run_results
    with the returned test_run_id to retrieve scores.

    Args:
        scenario_name: Name of the scenario to evaluate against.
        model_name: Name of the registered model to evaluate.
        checks: List of check names to apply (e.g., ["coherence", "relevance"]).
            Use list_checks to discover available checks and pick from the
            category matching the task and modality — do not use
            voice-specific checks for text evaluations (or vice versa);
            checks outside voice-specific categories suit both.
        name: Optional human-readable name for this test run.
        type: Type of evaluation. Defaults to NL_GENERATION. Valid values:
            NL_GENERATION, INFORMATION_RETRIEVAL, MULTI_CLASS_CLASSIFICATION,
            INVARIANT, MULTI_TURN, AGENT_EVAL.
    

Inputs: checks (required), model_name (required), scenario_name (required), name, +1 more. Returns: result

save_dashboardSave DashboardA
Idempotent
Inspect

Create or update an analytics dashboard by name (upsert).

    If a dashboard with this name already exists it is updated; otherwise a
    new one is created.

    Size each panel with a named ``size`` from the catalog below (PREFERRED
    — guarantees a legible layout) and omit positions entirely: panels are
    auto-placed in the order given (left-to-right, top-to-bottom on a
    12-column grid, wrapping rows, never overlapping).

    Size catalog and when to use each:

    - ``small-square`` (3x6): single ``stat`` metrics.
    - ``half-rectangle`` (6x6): ``line``/``bar``/``area`` trends, two per row.
    - ``half-square`` (6x9): ``radar``, ``composed``, denser charts.
    - ``full-rectangle`` (12x9): wide time-series comparisons.
    - ``full-square`` (12x12): ``table`` panels.

    Args:
        name: Dashboard name — the upsert key.
        panels: Optional list of panel definitions. Each panel is an object:

            - ``title`` (str, required): panel heading.
            - ``chart_type`` (str, required): one of ``line``, ``bar``,
              ``composed``, ``area``, ``radar``, ``stat``, ``table``.
            - ``query`` (object, required): what to chart —
              ``{"cube": "check_trend", "measures": [...],
              "dimensions": [...], "filters": [...],
              "time_dimensions": [...], "order": {...}}``. ``measures`` is
              required; everything else is optional. ``cube`` defaults to
              ``check_trend``. The dashboard ``time_range`` applies to all
              panels — panels do NOT carry their own time range.
            - ``size`` (str): a catalog name (see above). Required unless
              ``layout`` is given; wins over ``layout`` w/h if both appear.
            - ``layout`` (object): raw grid placement
              ``{"x": >=0, "y": >=0, "w": >=1, "h": >=1}`` (integers).
              Only needed when not using ``size``, or to pin an explicit
              position (give both ``x`` and ``y``; with ``size``, w/h are
              ignored). Heights below the legibility floor are sized up on
              save: ``h >= 6`` when ``w <= 6``, ``h >= 9`` when ``w > 6``.
            - ``table_config`` (object, optional): ONLY for
              ``chart_type == "table"``.

            ``check_trend`` measures: ``avg_check_value``, ``issue_rate``,
            ``error_rate``, ``datapoint_count``, ``issue_count``,
            ``error_count``, ``test_run_count``, ``avg_latency``,
            ``sum_cost``, ``input_token_count``, ``output_token_count``.
            ``check_trend`` dimensions: ``check.name``, ``check.id``,
            ``target.name``, ``target.id``, ``scenario.name``,
            ``scenario.id``, ``test_run.id``, ``test_run.type``,
            ``test_run.is_latest_for_target``, ``source``, ``provider``,
            ``request_model_name``, ``response_model_name``, ``tag``.
            Use ``query_analytics(include_metadata=True)`` for the
            authoritative, current set.

            Example panel::

                {"title": "Avg Check Value by Check", "chart_type": "bar",
                 "query": {"measures": ["avg_check_value"],
                           "dimensions": ["check.name"]},
                 "size": "half-rectangle"}
        description: Optional dashboard description.
        time_range: Optional default look-back window for the whole
            dashboard. One of: LAST_HOUR, LAST_24_HOURS, LAST_7_DAYS,
            LAST_14_DAYS, LAST_30_DAYS, LAST_90_DAYS. Defaults to
            LAST_90_DAYS when omitted.

    Returns:
        JSON with the saved dashboard and ``action`` (created/updated).
        When sizing or dimensions were changed on save (size overriding
        layout w/h, or a height floored), an ``adjustments`` list reports
        each change: ``{"panel", "field", "from", "to", "reason"}``.
    

Inputs: name (required), panels, time_range, description. Returns: result

save_scenarioSave ScenarioA
Idempotent
Inspect

Save a named scenario for use in quality tests.

        Provide EXACTLY ONE dataset source: `content`, `file_path`, or
        `rows`.

        Prefer `file_path` for local .jsonl files — the server reads the
        file directly, so no rows pass through the assistant's context.
        When passing rows through the assistant instead, keep the dataset
        UNDER 2,000 rows (`content` with the file's text, or `rows` for
        tiny datasets). For 2,000 rows or more, always use `file_path` or
        upload directly to Okareo via the web app, SDK, or CLI, to avoid
        unnecessary token cost.

        If a scenario with the same name already exists, the existing
        scenario is returned (idempotent). Scenarios are immutable after
        creation — use create_scenario_version to create updated versions.

        Args:
            name: A unique name for the scenario.
            content: Raw JSONL text (one JSON object with 'input' and
                'result' per line). Only for datasets under 2,000 rows.
            file_path: Path to a local .jsonl file. Preferred — works for
                any size.
            rows: List of data rows, each with 'input' (any type) and
                'result' (any type). Use for small scenarios (< 20 rows).
            tags: Optional list of tags for categorizing the scenario.
        

Inputs: name (required), rows, tags, content, +1 more. Returns: result

switch_tenantSwitch TenantB
Read-onlyIdempotent
Inspect

Change which Okareo organization your session operates against.

    Organization selection now happens **during sign-in** (feature 030):
    when you connect the Okareo MCP you choose which organization to
    authorize, and the credential this session uses is already scoped to
    it. This tool therefore no longer changes the active organization — to
    switch, reconnect/re-authenticate the Okareo MCP from your copilot and
    select a different organization when prompted. Use ``list_tenants`` to
    see which organization is currently active.
    

Inputs: tenant_id (required). Returns: result

update_generation_modelUpdate Generation ModelA
Idempotent
Inspect

Change the LLM that a registered generation model points to.

    Updates the generation model to use a different LLM from the registry.
    Use list_available_llms to see available LLMs.

    Args:
        name: Name of the registered generation model to update.
        model_name: The new LLM from the registry.
    

Inputs: name (required), model_name (required). Returns: result

update_voice_integrationUpdate Voice IntegrationA
Idempotent
Inspect

Update a voice provider integration's metadata.

    Args:
        integration_id: The integration's id.
        metadata: The new metadata object.
    

Inputs: metadata (required), integration_id (required). Returns: result

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

Loading tool schema changelog…

    TDQS

    A3.7/5.0

    Scored across 51 tools

    Disambiguation4/5

    Most tools are clearly distinct by resource and action, with only a few potentially confusing pairs like run_test vs run_simulation or register_generation_model vs create_or_update_target with a 'generation' type. Detailed descriptions and distinct parameters largely resolve ambiguity, so an informed agent can reliably select the right tool.

    Naming Consistency4/5

    The set predominantly follows a verb_noun pattern (list_, get_, create_or_update_, delete_) with consistent snake_case. Minor deviations exist, such as save_scenario/save_dashboard instead of create_, register_generation_model instead of create_generation_model, and longer phrases like get_voice_webhook_url, which keep it from being perfectly uniform.

    Tool Count2/5

    At 51 tools, the server is well beyond the 25+ threshold considered too many for a single MCP server. While the broad scope of the Okareo platform explains the abundance, the sheer size will overwhelm agents and make tool selection and discovery unnecessarily difficult.

    Completeness3/5

    The server covers a wide range of resources with robust CRUD support, but there are notable gaps: there is no delete_driver tool, and no way to delete test runs. The driver lifecycle in particular is incomplete, which is a significant omission for an otherwise comprehensive platform surface.

    Maintenance

    ActivityMaintained
    ResponsivenessNo recent community issues

    Related MCP Connectors

    Related MCP Servers