Skip to main content
Glama
okareo-ai

Okareo MCP Server

Official
by okareo-ai

Okareo MCP Server

The Okareo MCP server exposes Okareo's evaluation capabilities as MCP tools, allowing AI coding assistants to create and manage scenarios, register models, run evaluations, and execute multi-turn simulations directly from your editor.

For detailed documentation, see the Okareo MCP docs.

Prerequisites

  • An Okareo account at app.okareo.com

  • A copilot that supports MCP servers (Claude Code, Cursor, or VS Code)

  • Python 3.10–3.12 (only for the local install modes; not needed for remote)


Related MCP server: Patronus MCP Server

The fastest way to get started. No local install, no Python, no API key in .mcp.json. Browser sign-in handles auth on first connect.

Add to your copilot's MCP config (typically .mcp.json):

{
  "mcpServers": {
    "okareo": {
      "type": "http",
      "url": "https://tools.okareo.com/mcp"
    }
  }
}

Reload the copilot. It will open a browser to Okareo sign-in once; thereafter the copilot stores the token itself. Your .mcp.json contains no secrets.

Fallback (Bearer header — older clients or headless / CI)

For clients that haven't shipped the MCP OAuth flow yet, paste your API key as a bearer header. Prefer the env-var form over an inline literal:

{
  "mcpServers": {
    "okareo": {
      "type": "http",
      "url": "https://tools.okareo.com/mcp",
      "headers": {
        "Authorization": "Bearer ${env:OKAREO_API_KEY}"
      }
    }
  }
}

Working across multiple Okareo organizations

If you have access to more than one Okareo account, simply Clear Authentication and then re-authenticate with the account you want to work in. You can always ask which account your are in and have acccess to.


Local install (alternative)

Run the MCP server on your own machine. Useful for offline / airgapped environments and for development.

Prefer the Remote MCP section above unless you have a specific reason to install locally (airgapped, custom build, development on this repo). The remote endpoint requires no install and stays current automatically.

For multi-org users, the remote endpoint also exposes tenant management tools (list_tenants, switch_tenant).

Step 1: Set Your API Key

export OKAREO_API_KEY="your-api-key"

Add this to your ~/.zshrc or ~/.bash_profile for persistence.

Step 2: Configure Your Copilot

Claude Code

Add to .mcp.json:

{
  "mcpServers": {
    "okareo": {
      "command": "uvx",
      "args": ["okareo-mcp"],
      "env": {
        "OKAREO_API_KEY": "${OKAREO_API_KEY}"
      }
    }
  }
}

No pre-install needed — uvx handles it automatically.

Cursor

Add to .cursor/mcp.json:

{
  "mcpServers": {
    "okareo": {
      "command": "uvx",
      "args": ["okareo-mcp"]
    }
  }
}

Cursor inherits OKAREO_API_KEY from your shell environment.

Alternative: pip install

If you don't have uv installed:

pip install okareo-mcp

Then use "command": "okareo-mcp" instead of "command": "uvx" with "args": ["okareo-mcp"].


Configuration Reference

Variable

Default

Description

OKAREO_API_KEY

(required)

Your Okareo API key

OKAREO_BASE_URL

https://api.okareo.com

Override for on-prem Okareo backend

TRANSPORT

stdio

Transport: stdio (local) or sse (Docker)

PORT

8000

Port for SSE transport


Available Tools

Scenarios

Tool

Description

save_scenario

Save a named scenario from rows of input/result data (idempotent)

list_scenarios

List all scenarios in the project with names, IDs, and row counts

get_scenario

Retrieve a scenario's metadata and all data rows by name or ID

create_scenario_version

Create a new version of an existing scenario with updated data

preview_delete_scenario

Preview what will be deleted before removing a scenario

delete_scenario

Permanently delete a scenario and all related test data

Generation Models

Tool

Description

list_available_llms

Browse available LLMs from the Okareo registry

register_generation_model

Register a generation model for testing by selecting an LLM from the registry

list_generation_models

List all registered generation models in the project

get_generation_model

Read detailed information about a registered generation model

update_generation_model

Change the LLM a registered generation model points to

delete_generation_model

Remove a registered generation model and all its related test data

Tests & Checks

Tool

Description

list_checks

List available quality checks (built-in and custom) for evaluating model outputs

run_test

Run a quality test that evaluates a model against a scenario using specified checks

list_test_runs

List past test runs with optional filters (model, scenario, simulation-only)

get_test_run_results

Load detailed per-row results of a test run or simulation by ID or name

get_conversation_transcript

Retrieve the full conversation transcript for a single data point

reevaluate_test_run

Re-score a completed test run against a (possibly different) set of checks

create_or_update_check

Create or update a quality check by name — model-based, code-based, or audio (upsert)

generate_check

Generate a check from a natural-language description, then save it

get_check

Retrieve a check's full configuration, including its prompt template or code

delete_check

Permanently delete a check by name

calibrate_check

Dry-run a draft check against a finished test run — per-row verdicts and the arguments the check received, saving nothing

Simulations (Multi-Turn)

Tool

Description

create_or_update_target

Create or update a Target — generation model, custom endpoint, or voice (OpenAI, Deepgram, Twilio)

get_target

Retrieve a Target's configuration by name (all types)

list_targets

List all simulation targets (voice and custom_endpoint) in the project

delete_target

Remove a simulation target and all its related test data

create_or_update_driver

Define a simulated user persona that will interact with your target

get_driver

Retrieve a Driver's full configuration including the persona prompt

list_drivers

List all Driver personas in the project

list_driver_voices

Discover the voices, voice profiles, and languages available for voice drivers

run_simulation

Run a multi-turn conversation evaluation (or rerun a previous one with overrides)

list_simulations

List past simulation runs with optional filters (target, scenario, limit)

Voice Monitoring

Tool

Description

ingest_conversations

Submit completed voice conversations to Okareo for monitoring

connect_voice_integration

Connect a voice provider so its traffic flows into Okareo monitoring

list_voice_integrations

List the voice provider integrations in your project

get_voice_integration

Retrieve a voice provider integration by ID, including its status

update_voice_integration

Update a voice provider integration's metadata

rotate_voice_integration_secret

Rotate a voice provider integration's secrets

delete_voice_integration

Delete a voice provider integration by ID

get_voice_webhook_url

Get the inbound webhook endpoint for a voice provider

Projects

Tool

Description

list_projects

List every project you can work in — id, name, tags, archive state — with the active one marked

select_project

Choose the project to work in for this conversation

create_project

Create a new project

update_project

Rename a project, or replace its tags

archive_project

Hide a project from the project picker. Reversible; nothing is deleted

unarchive_project

Restore an archived project to the picker

clone_project

Copy an existing project's scenarios into a new project (dry-run first)

Okareo has no project delete — in the app, the API, or here. Archiving is the removal, and it only hides the project from the picker: its scenarios, runs, and dashboards stay intact and every tool still works against it.

Analytics & Dashboards

Tool

Description

query_analytics

Query Okareo's product analytics to understand evaluation trends

list_dashboards

List the analytics dashboards in your project

get_dashboard

Retrieve a dashboard's full configuration by name

save_dashboard

Create or update an analytics dashboard by name (upsert)

reorder_dashboards

Set the display order of dashboards

delete_dashboard

Delete a dashboard by name

Tenant Management (remote MCP only)

Tool

Description

list_tenants

List every Okareo organization you have access to in this MCP session

switch_tenant

Change which Okareo organization subsequent tool calls operate against

Documentation & Templates

Tool

Description

get_docs

Query the Okareo documentation system for conceptual or user-legible explanations

get_templates

Retrieve prompt templates for common Okareo patterns (works offline)

get_reps_baseline

Serve REPS agent-evaluation baseline material (scenario banks, drivers, checks, eval configs) from the latest tagged okareo-tools release


Troubleshooting

Symptom

Cause

Fix

okareo-mcp: command not found

Not installed or not in PATH

Run pip install -e . (dev) or use uvx okareo-mcp (user)

Server exits with API key error

OKAREO_API_KEY not set

Export it: export OKAREO_API_KEY="..."

pip install fails on Python 3.13+

Okareo SDK requires Python <3.13

Use Python 3.10–3.12

Copilot can't connect (Docker)

Wrong URL

Ensure URL ends with /sse and port matches

Cursor doesn't pick up API key

Cursor launched from Dock, not terminal

Launch Cursor from terminal: cursor .


Contributing

This repository is a curated public mirror; the canonical source is maintained by Okareo. We welcome issues and consider community pull requests — see CONTRIBUTING.md for how proposed changes are reviewed and ported.

License & Trademarks

The Okareo MCP server source code is licensed under the Apache License 2.0 (see also NOTICE).

"Okareo", the Okareo logo, and Okareo product names are trademarks of Okareo, Inc. and are not covered by the Apache 2.0 license. See TRADEMARK.md for permitted use.

Available Tools

51 tools
connect_voice_integrationConnect Voice IntegrationAInspect

Connect a voice provider so its traffic flows into Okareo monitoring.

    Creates a provider integration. The returned integration carries an id
    and a public_id — pass the provider + public_id to get_voice_webhook_url
    to obtain the inbound webhook endpoint to paste into the provider's
    console.

    Args:
        provider: Voice platform — one of: retell, twilio, vapi, elevenlabs.
        webhook_auth_type: Webhook authentication type expected by Okareo
            for this provider (provider-specific — see Okareo docs).
        secrets: Provider-specific secret values (opaque pass-through; the
            response never echoes raw secrets, only a summary).
        metadata: Optional free-form metadata object.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
secretsYes
metadataNo
providerYes
webhook_auth_typeYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations, the description adds meaningful behavioral details: the response carries an id and public_id, secrets are never echoed and only summarized, and webhook_auth_type is provider-specific. This helps the agent understand side effects and security aspects. No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is about 150 words and well-structured, with a clear purpose statement followed by parameter explanations. Every sentence adds value; no fluff or repetition. It's slightly longer than necessary but appropriate given the parameter complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description covers the main success behavior (creation, returned id/public_id) and directs to the next step (get_voice_webhook_url). With an output schema present, it doesn't need to enumerate all return fields. It also covers all parameters and provider options, making it sufficiently complete for this mutating tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 0% description coverage, so the Args section in the description carries the full burden. It adds substantial meaning: lists allowed provider values (retell, twilio, vapi, elevenlabs), explains webhook_auth_type is provider-specific, describes secrets as opaque pass-through, and notes metadata is optional. This fully compensates for the missing schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's action: "Connect a voice provider so its traffic flows into Okareo monitoring" and "Creates a provider integration." This is a specific verb+resource, and it distinguishes from sibling tools like update_voice_integration or rotate_voice_integration_secret by focusing on initial setup.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for when to use it (connecting a voice provider) and even mentions the follow-up step of using get_voice_webhook_url with the returned public_id. It doesn't explicitly state exclusions or alternatives, but the focus on creation makes the usage fairly obvious.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

create_or_update_checkCreate or Update CheckA
Idempotent
Inspect

Create or update a quality check by name (upsert). Supports model-based, code-based, and audio checks.

    Saving to an existing name creates a new version of that check (see
    get_check's "available_versions"). Before writing a prompt_template or
    code_contents from scratch, fetch a worked example with get_templates:
    "boolean_check_prompt", "score_check_prompt", "analysis_check_prompt",
    or "check_code".

    Args:
        name: Unique name for the check.
        description: What the check evaluates.
        check_type: "model" (an LLM judge driven by prompt_template) or
            "code" (a deterministic Python class in code_contents).
        output_type: "pass_fail" (boolean verdict), "score" (numeric, e.g.
            a 1-5 rubric), or "analysis" (free-form qualitative feedback;
            only valid with check_type="model"). For check_type="code" the
            server infers pass_fail vs score from the value evaluate()
            returns (bool vs int/float) — output_type is used only to
            validate the request, not sent to the server. Note: list_checks
            and get_check report this as output_data_type in the server
            vocabulary, where "bool" means pass_fail and "int" means score.
        prompt_template: Required when check_type="model". The judge
            prompt. Inject the runtime data the judge needs with these
            placeholders:
            - {model_output}: the model output being evaluated. In a
              multi-turn conversation this is ONLY the final assistant
              message, not the full conversation.
            - {scenario_input}: the scenario input / source text.
            - {scenario_result}: the reference/expected output.
            - {model_input}: what was sent to the model (prompt or
              messages).
            - {message_history}: the full multi-turn conversation — the
              model_input messages plus the assistant's model_output. Use
              this when the check must judge the whole conversation.
            - {tool_calls}: the tool/function calls the model just made.
            - {tools}: the tool definitions/schema available to the model.
            - {model_output_metadata}: metadata attached to the most
              recent model output.
            - {simulation_message_history}: full conversation history
              reconstructed from trace metadata. Only populated for traced
              (ingested) conversations; for simulations and evaluations
              use {message_history}.
            The legacy {generation} placeholder is deprecated — use
            {model_output} instead.
        code_contents: Required when check_type="code" (output_type
            "pass_fail" or "score" only). Python source defining
            `class Check(CodeBasedCheck)` with a
            `@staticmethod def evaluate(...) -> CheckResponse` method.
            Start from `from okareo.checks import CodeBasedCheck,
            CheckResponse`. evaluate() may declare any subset of these
            parameters: model_output, scenario_input, scenario_result,
            metadata, model_input. Return CheckResponse(score=...,
            explanation=...) where score is a bool for pass_fail or an
            int/float for score. See get_templates("check_code") for
            complete examples.
        is_audio: Set to true for audio/voice evaluation. Only valid with
            check_type="model".
        tags: Optional list of string tags to organize the check. Tags are
            stored with the check and returned by get_check.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes
tagsNo
is_audioNo
check_typeYes
descriptionYes
output_typeYes
code_contentsNo
prompt_templateNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

While annotations already indicate idempotency (idempotentHint=true), the description adds crucial behavioral nuances: saving to an existing name creates a new version, output_type for code checks is only validated client-side and not sent to the server, and the server-side vocabulary difference (output_data_type) is explained. It also notes the deprecated placeholder, providing transparency beyond what annotations offer.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but exceptionally well-structured. It leads with the main purpose, then organizes parameter details in an 'Args' block with clear formatting. The extensive placeholder list and code examples are essential for a tool with this complexity, and every section carries meaningful information without repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (8 parameters, model vs code check variants, audio support, placeholder injection), the description is remarkably complete. It covers prerequisites (get_templates), versioning behavior, parameter interdependencies, code class structure, and server-side nuances. Since an output schema is present, the absence of return-value explanation is acceptable.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, but the description fully compensates by explaining each parameter in depth. It details valid values, dependencies (e.g., prompt_template required for model checks), placeholder semantics for prompt_template, and the expected Python structure for code_contents. This far exceeds the schema's bare type definitions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'Create or update a quality check by name (upsert).' It clearly distinguishes from siblings like get_check, delete_check, and generate_check by focusing on the creation/update operation. The scope (model, code, audio) is explicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for when to use this tool and when to consult alternatives. It explicitly directs users to get_templates for worked examples before writing prompt_template or code_contents from scratch, and references get_check's 'available_versions' for versioning behavior. It does not explicitly exclude alternative tools like generate_check, but the contextual guidance is strong.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

create_or_update_driverCreate or Update DriverA
Idempotent
Inspect

Define a simulated user persona that will interact with your target.

    Creates or updates a Driver by name (upsert). Author ONLY the core
    persona sections in prompt_template, in this order:

    - `## Persona` — who the simulated user is (static character).
    - `## Scenario Details` — contains the scenario reference
      (`{scenario_input}` or a specific path like
      `{scenario_input.objectives}`), placed immediately before
      Objectives. This is how each scenario row's data reaches the
      conversation.
    - `## Objectives` — WHAT the driver is trying to accomplish, written
      from the driver's goal (not from scenario variables).
    - `## Soft Tactics` — HOW the driver probes, escalates, and stops.

    Do NOT author Hard Rules, a Turn-End Checklist, or Conversation
    Behavior sections: the MCP automatically appends the platform's
    canonical versions of those blocks (including the language rule
    matching `language`) — the same blocks the Okareo UI appends to
    generated drivers. Any caller-authored variant of these sections is
    replaced by the canonical text, and repeated updates never duplicate
    the blocks.

    For voice agents, configure how the simulated user speaks with `voice`,
    `voice_profile`, `voice_instructions`, and `language`. Call
    list_driver_voices first to discover valid voice and profile values.

    Args:
        name: Unique name for this driver.
        prompt_template: The core persona prompt (Persona, Scenario
            Details, Objectives, Soft Tactics — see above). Hard Rules and
            Conversation Behavior are appended automatically.
        model_id: Foundation model to power the driver (defaults to project default).
        temperature: Response randomness, default 0.6.
        voice_instructions: Free-text speaking instructions for voice simulations
            (tone, pace, accent). Not validated against the voice catalog.
        voice_profile: Voice profile name for voice simulations. Validated
            against the catalog from list_driver_voices.
        voice: Voice identifier for voice simulations. Validated against the
            catalog from list_driver_voices.
        language: Language the driver responds in, as the bare ISO code
            the voice catalog serves (e.g. "en", "es", "ja"); regional
            variants like "fr-CA" are accepted when their base code
            matches the voice's language. When a `voice` is set and
            language is omitted, it is derived from that voice's catalog
            language (disclosed as `language_derived_from_voice` in the
            response); a value conflicting with the voice's language is
            rejected. Also drives the appended Hard Rules language rule.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes
voiceNo
languageNo
model_idNo
temperatureNo
voice_profileNo
prompt_templateYes
voice_instructionsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations (idempotentHint=true), the description discloses that caller-authored canonical sections are replaced and repeated updates never duplicate blocks. It also reveals language conflict rejection and language derivation behavior with voice, which are non-obvious behavioral traits not inferable from annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is organized with an opening summary, explicit authoring rules, voice configuration guidance, and a per-parameter breakdown. Every sentence adds value, and the structure makes scanning easy despite the length.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description is fully complete for a complex tool: it covers all parameters, prerequisites, validation behavior, and automatic append behavior. It even mentions response disclosure (language_derived_from_voice). With an output schema available, no return description is needed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The 'Args:' section adds rich semantics for all 8 parameters: uniqueness of name, allowed prompt_template sections, model_id default, temperature default, voice validation against the catalog, and language rules including regional variants and derivation. This goes far beyond the bare schema types and defaults.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a clear summary: 'Define a simulated user persona that will interact with your target. Creates or updates a Driver by name (upsert).' This is a specific verb+resource statement that distinguishes the tool from siblings like get_driver, list_drivers, and list_driver_voices.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives explicit usage guidance, including a 'Do NOT author Hard Rules...' warning and instructs callers to 'Call list_driver_voices first to discover valid voice and profile values.' This tells when to use a sibling tool first and what not to include, which is explicit when/when-not guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

create_or_update_targetCreate or Update TargetA
Idempotent
Inspect

Create or update a Target — the AI system you want to evaluate in a simulation.

    Calling create_or_update_target with the same name as an existing Target will
    **fully replace** its configuration — caller must re-specify all desired fields.
    Supported types: 'generation' (foundation model), 'custom_endpoint' (your own
    REST API), and 'voice' (voice-based targets reached by phone via Twilio or
    over SIP).

    **Cloning workflow**: this tool accepts the same key structure that `get_target`
    returns, so you can read an existing Target, change `name`, swap in real values
    for any field whose value is `"***REDACTED***"`, and pass the result here as
    kwargs. Calls that still contain the redaction sentinel are rejected with an
    error naming each offending path; the sentinel is never forwarded to the backend.

    Args:
        name: Unique name for this target.
        type: Target type — 'generation', 'custom_endpoint', or 'voice'.

        model_id: (generation targets) Foundation model ID, e.g. 'gpt-4o-mini'.
        temperature: (generation targets) Response randomness, default 0.
        system_prompt_template: (generation targets) System instructions; mustache
            syntax supported, e.g. '{scenario_input}'.
        user_prompt_template: (generation targets) User prompt template.
        dialog_template: (generation targets) Dialog formatting template.
        tools: (generation targets) Tool definitions for function calling.

        next_message_params: (custom_endpoint) Nested HTTP config for each
            conversation turn. Required keys: 'url', 'method'. Optional:
            'headers', 'body', 'status_code', 'response_message_path',
            'response_session_id_path', 'response_tool_calls_path'.
            All response path values MUST use dot-path notation starting with
            'response.' — e.g., 'response.message', 'response.choices[0].message.content',
            'response.choices[0].tool_calls'. Never use bare property names.
            For SSE/streaming endpoints, include a 'streaming' object with:
              - 'stop': array of stop conditions (OR semantics — any match ends
                the stream). Each has 'value' (required) and optional 'path'
                (dot-path into JSON chunk). Without 'path', matches raw SSE data.
              - 'select': array of select conditions (AND semantics — all must
                match for a chunk's content to be extracted). Each requires
                'path' and 'value'.
            When streaming, set response_message_path to the chunk field
            (e.g., 'response.choices[0].delta.content').
        start_session_params: (custom_endpoint, optional) Nested HTTP config to
            initialise a session. Required key: 'url'. Optional: 'method',
            'headers', 'body', 'status_code', 'response_session_id_path'
            (dot-path starting with 'response.', e.g. 'response.id'),
            'response_message_path'. Supports 'streaming' object (same
            structure as next_message_params.streaming).
        end_session_params: (custom_endpoint, optional) Nested HTTP config to
            close a session after the last turn.
        auth_params: (custom_endpoint, optional) Token-based authorization config.
            Required keys when provided: 'url', 'method', 'response_access_token_path'
            (dot-path starting with 'response.', e.g. 'response.access_token').
            Optional: 'headers', 'body', 'status_code'.
        sensitive_fields: (custom_endpoint, optional) List of dot-path strings for
            secret fields within auth_params (e.g., 'auth_params.body.client_id').
            The MCP auto-generates entries for top-level auth_params keys; use this
            for deeper paths. To remove auth from an existing target, call
            create_or_update_target again without auth_params.
        max_parallel_requests: (custom_endpoint, twilio) Concurrency limit. This is
            the same setting the Okareo web UI labels "max concurrency".

        edge_type: (voice targets) How Okareo reaches the voice agent —
            'twilio' (dial a phone number) or 'sip' (call a SIP URI).

        to_phone_number: (voice twilio) Destination phone number (required).
        account_sid: (voice twilio, custom only) Twilio account SID. If provided,
            auth_token and from_phone_number are also required (all-or-nothing).
            Omit for generic Twilio targets using Okareo's managed integration.
        auth_token: (voice twilio, custom only) Twilio auth token. Required with
            account_sid and from_phone_number.
        from_phone_number: (voice twilio, custom only) Caller phone number. Required
            with account_sid and auth_token.

        sip_uri: (voice sip) Destination SIP URI (required), e.g.
            "sip:agent@your-domain.example.com". Use this to test any voice
            agent reachable over SIP — for example one fronted by Daily,
            Vapi, LiveKit, or a SIP trunk.
        sip_username: (voice sip, optional) SIP authentication username.
        sip_password: (voice sip, optional) SIP authentication password
            (stored as a sensitive field).
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes
typeYes
toolsNo
sip_uriNo
model_idNo
edge_typeNo
auth_tokenNo
account_sidNo
auth_paramsNo
temperatureNo
sip_passwordNo
sip_usernameNo
dialog_templateNo
to_phone_numberNo
sensitive_fieldsNo
from_phone_numberNo
end_session_paramsNo
next_message_paramsNo
start_session_paramsNo
user_prompt_templateNo
max_parallel_requestsNo
system_prompt_templateNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations (idempotentHint=true, destructiveHint=false), the description discloses critical behaviors: 'fully replace its configuration — caller must re-specify all desired fields', redaction sentinel rejection, auto-generation of sensitive fields, and the all-or-nothing rule for Twilio auth. These details significantly exceed what the annotations alone convey.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but justified by the tool's complexity (22 parameters, three target types). It uses clear sectioning (type-specific parameter groups) and front-loads the core purpose and replacement warning. A few sentences, like the streaming stop/select details, are dense but necessary. Very little is wasted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with 22 parameters, 0% schema descriptions, and a rich output schema, the description is fully complete. It covers all parameter categories, type-specific configurations, redaction handling, replacement semantics, and auth removal. The presence of an output schema means return-value documentation is unnecessary, and the description fills all other gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema coverage, the description carries the full burden and excels: every one of the 22 parameters is documented with type-specific context, requiredness, defaults, examples, and nested structures (e.g., next_message_params with dot-path rules and streaming stop/select semantics). This adds enormous meaning beyond the bare schema property names.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource: 'Create or update a Target — the AI system you want to evaluate in a simulation.' It clearly distinguishes from sibling tools like get_target, list_targets, and delete_target by covering both creation and update in one operation. The title and description align perfectly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear usage context, including the cloning workflow ('this tool accepts the same key structure that `get_target` returns') and the replacement behavior when reusing a name. It also explains type-specific requirements (generation, custom_endpoint, voice) and how to remove auth. However, it does not explicitly state when to use an alternative tool, instead relying on the context of the cloning workflow.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

create_scenario_versionCreate Scenario VersionAInspect

Create a new version of an existing scenario with updated data.

    Automatically determines the next version number (e.g., 'my-test-v2',
    'my-test-v3'). The original scenario is treated as version 1.

    Args:
        base_name: Name of the original scenario to create a version of.
        rows: List of data rows for the new version, each with 'input' and 'result'.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
rowsYes
base_nameYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.8/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only indicate it is a non-read-only, non-idempotent, non-destructive operation. The description adds useful behavioral details: it automatically determines the next version number (e.g., 'my-test-v2'), treats the original as version 1, and specifies the row structure. However, it does not disclose error behavior (e.g., if base_name does not exist) or side effects beyond creation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is efficiently structured: a clear one-line summary, followed by a concise explanation of version numbering and an Args list. The Args section is useful and not overly verbose. It earns its place, though the docstring format is slightly technical.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool is simple with two required parameters and an output schema, so the description covers the essentials: purpose, versioning behavior, and parameter semantics. However, it lacks usage guidelines relative to alternatives and does not address prerequisites or error cases, leaving some contextual gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description compensates by explaining both parameters: base_name is the original scenario name, and rows are data rows with 'input' and 'result' fields. This adds meaning beyond the schema's bare titles and types, though it could further clarify required field types or whether rows must contain exactly those keys.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: 'Create a new version of an existing scenario with updated data.' It uses a specific verb+resource combination and distinguishes it from siblings like save_scenario by emphasizing versioning and automatic version-number determination.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage context (creating new versions with auto-incremented numbering) but does not explicitly state when to use it over alternatives like save_scenario, nor does it provide exclusions or conditions. The behavior of generating the next version number is a clear signal, but no direct usage guidance is given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

delete_checkDelete CheckA
Destructive
Inspect

Permanently delete a check by name.

    Args:
        name: Name of the check to delete.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds the word 'permanently', which is a behavioral trait not explicitly stated in the annotations. However, it does not disclose any other side effects, failure modes, prerequisites, or what happens to associated data. The destructiveHint annotation already signals destructiveness, so the description's contribution is minimal but present.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is extremely concise, consisting of a single clear sentence and an Args block. Every element is necessary and there is no wasted text. It is well-structured and front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple single-parameter destructive operation with annotations indicating destructiveness and an output schema, the description is adequate. The 'permanently' keyword signals irreversibility, and the parameter is explained. It doesn't detail return values, but the output schema likely covers that. Given the tool's simplicity, the description is nearly complete, though it could mention error conditions (e.g., check not found).

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has no description for the 'name' parameter (0% coverage), so the description provides the meaning by stating 'Name of the check to delete'. This clarifies the parameter's role, but it does not specify whether the name is a unique identifier, case-sensitive, or how it relates to names in other tools. The description compensates for the schema gap but only at a basic level.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb 'delete' with a specific resource 'check' and adds 'permanently' to clarify the action's scope. It clearly distinguishes this from sibling tools like get_check, list_checks, and create_or_update_check.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no explicit guidance on when to use this tool versus alternatives. It does not mention checking existence first, using preview_delete_scenario for checks, or any when-not-to-use conditions. The usage is only implied by the verb 'delete'.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

delete_dashboardDelete DashboardB
Destructive
Inspect

Delete a dashboard by name.

    Args:
        name: Name of the dashboard to delete.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

B3.1/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotation already declares destructiveHint=true. The description adds no extra behavioral context such as permanence, irreversibility, or required permissions. It simply restates the basic operation without enriching the safety profile.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is very short and front-loaded with the main action. It avoids unnecessary verbiage, though it could have been slightly more informative without becoming verbose.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the single parameter and the destructive annotation, the description is minimally sufficient but lacks context about error handling (e.g., what happens if the dashboard doesn't exist) or any side effects. For a destructive operation, slightly more detail would improve completeness.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has zero description coverage, and the description only repeats the parameter name ('name: Name of the dashboard to delete'). It adds no additional meaning about format, uniqueness, or edge cases beyond what the schema field name already implies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Delete') and the specific resource ('a dashboard') with a required parameter ('by name'). This distinguishes it from sibling delete tools that target other resources (e.g., delete_scenario, delete_generation_model).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is provided about when to use this tool versus alternatives. It does not mention any prerequisites, exclusions, or contextual conditions for deletion.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

delete_generation_modelDelete Generation ModelA
Destructive
Inspect

Remove a registered generation model and all its related test data.

    Permanently deletes the generation model and cascades to associated
    test runs and test data points. This cannot be undone.

    Args:
        name: Name of the registered generation model to delete.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description goes beyond the annotation's destructiveHint by detailing specific behavioral traits: it 'cascades to associated test runs and test data points' and is irreversible ('This cannot be undone'). This adds meaningful context about the destructive scope that the annotation alone does not convey.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and front-loaded with the action. It includes two brief sentences and an Args list, all of which add value. It could be slightly tighter by merging the first two sentences, but there is no waste and the structure is clear.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple delete tool with one required parameter, the description is nearly complete. It covers the action, cascading effects, irreversibility, and the parameter meaning. The presence of an output schema and annotations reduces the need to describe return values or safety flags, making this sufficiently comprehensive.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema only provides the parameter name 'Name' with no description. The description's Args section supplies semantic meaning: 'Name of the registered generation model to delete.' This clarifies exactly what value is expected, fully compensating for the 0% schema description coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly identifies the tool's purpose: 'Remove a registered generation model and all its related test data.' It uses a specific verb ('remove') and resource ('generation model'), and explicitly mentions the cascading scope. This distinguishes it from sibling tools like update_generation_model or get_generation_model.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies the use case (deleting a model) and includes a strong caution ('This cannot be undone'), but it does not explicitly state when to use this versus alternatives, nor does it mention any prerequisites like listing models first. There are no exclusions or when-not-to-use guidance, so it is adequate but not highly instructive.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

delete_scenarioDelete ScenarioA
Destructive
Inspect

Permanently delete a scenario and all related test data.

    Both scenario_id and name are required. Use preview_delete_scenario first
    to see what will be deleted before confirming.

    Args:
        scenario_id: The ID of the scenario to delete (from preview_delete_scenario).
        name: The name of the scenario to delete.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes
scenario_idYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already mark destructiveHint=true, and the description enhances this by stating 'permanently delete' and 'all related test data', clarifying the scope and irreversibility. It also adds the safety practice of previewing, going beyond the structured annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is short, front-loaded with the core action, and presents usage guidance and parameter details efficiently without redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a destructive tool with an output schema, the description sufficiently covers prerequisites, the destructive scope, and the recommended preview step, making it complete for its complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema provides basic names/types, and the description adds context for scenario_id (from preview_delete_scenario) and reiterates that both are required. With 0% schema coverage, it partially compensates but could offer more detail on how to obtain or validate the name.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses the specific verb 'delete' with the resource 'scenario' and adds scope ('all related test data'), clearly distinguishing it from sibling tools like preview_delete_scenario and other delete_* tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly instructs to use preview_delete_scenario first before confirming deletion, providing direct guidance on proper tool sequencing and when to invoke this tool.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

delete_targetDelete TargetA
Destructive
Inspect

Remove a simulation target and all its related test data.

    Permanently deletes the target and cascades to associated
    test runs and test data points. This cannot be undone.

    Args:
        name: Name of the target to delete.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds that deletion is permanent, cascades to associated test runs and data points, and cannot be undone. This exceeds the destructiveHint annotation by detailing the scope and irreversibility, providing valuable behavioral context.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is three focused sentences plus a minimal Args block. It front-loads the main action, then adds cascade and irreversibility details, with no redundant text.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple one-parameter delete with an output schema and destructive annotation, the description covers the essential information: what is deleted, cascading behavior, and permanence. It does not mention error behavior for non-existent targets, but the annotation and low complexity make the description adequately complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description's Args section ('name: Name of the target to delete') provides the only explanation of the parameter, clarifying it is the target's name. However, it lacks additional constraints like uniqueness or how to obtain the name, but for a single-parameter delete tool it is sufficient.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states 'Remove a simulation target and all its related test data' with a clear verb and resource, and further clarifies permanent deletion and cascading to test runs/data points. This distinguishes it from other delete_ siblings by specifying the resource type (simulation target) and side effects.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for what the tool does (cascading deletion of a target) but does not explicitly state when to use it versus alternatives like delete_scenario or preview_delete_scenario. No exclusions or alternative recommendations are given, so usage is implied rather than explicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

delete_voice_integrationDelete Voice IntegrationB
Destructive
Inspect

Delete a voice provider integration by id.

    Args:
        integration_id: The integration's id.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
integration_idYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

B3.2/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds no behavioral detail beyond what the annotations already disclose (destructiveHint=true, readOnlyHint=false). It does not mention permanence, cascading effects, or potential failures, so it contributes no extra transparency.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is very concise and front-loaded with the core action. The Args block duplicates schema information unnecessarily, but the overall structure is efficient and easy to parse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the simple one-parameter operation, the description is adequate. Annotations and output schema cover the destructiveness and return value, but the description lacks context on side effects or when deletion is appropriate. It fits the tool's low complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With schema description coverage at 0%, the description must compensate. It provides a minimal definition ('The integration's id'), which clarifies the parameter's role but adds little beyond the schema's title. This is minimally sufficient for a single simple parameter.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Delete'), the resource ('voice provider integration'), and the scope ('by id'). This distinguishes it from sibling tools like update_voice_integration or list_voice_integrations, which perform different operations on the same resource.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no explicit guidance on when to use this tool versus alternatives (e.g., update_voice_integration) or any prerequisites/exclusions. It only implies usage for deletion, leaving the decision solely to the agent's intent.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

generate_checkGenerate CheckAInspect

Generate a check from a natural language description. Uses AI to create the prompt template (model checks) or Python code (code checks), then saves the check.

    Use this when you only have a description of what to evaluate. When
    you already know the exact prompt template or Python code the check
    should use, call create_or_update_check directly instead. The
    generated prompt/code is returned in the response — review it and
    refine with create_or_update_check if needed.

    Args:
        name: Name for the generated check.
        description: Natural language description of what to evaluate
            (e.g., "check if the response is toxic"). The more specific
            the description, the better the generated check.
        output_type: "pass_fail" (boolean verdict), "score" (numeric), or
            "analysis" (free-form qualitative feedback; model checks only).
        check_type: "model" (LLM judge) or "code" (deterministic Python).
        requires_scenario_input: Set true when the evaluation must compare
            the output against the scenario input. The generated check
            will reference {scenario_input} and only works on runs whose
            scenarios provide it.
        requires_scenario_result: Set true when the evaluation must
            compare the output against the expected result. The generated
            check will reference {scenario_result} and only works on runs
            whose scenarios provide it.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes
check_typeNomodel
descriptionYes
output_typeNopass_fail
requires_scenario_inputNo
requires_scenario_resultNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations show readOnlyHint=false, indicating a mutation. The description adds that AI generates the prompt/code, that the generated output is returned for review, and that scenario flags affect which runs the check works on. This goes beyond the bare annotations meaningfully, though it doesn't mention permissions or failure modes.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is somewhat lengthy but well-structured: a clear purpose statement, then usage guidance, then a structured argument list. Every section adds necessary value, especially given the low schema coverage. It could be slightly tightened, but the length is justified by the tool's complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description thoroughly covers the tool's purpose, when to use it, all parameters, the generated output returned for review, and dependencies on scenario flags. The presence of an output schema further reduces the need to describe return structure. It also points to create_or_update_check for refinement, providing a complete workflow.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description fully compensates. It explains all six parameters, including the allowed values for output_type ('pass_fail', 'score', 'analysis') and check_type ('model', 'code'), and clarifies the behavior of requires_scenario_input/result with example usage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool generates a check from a natural language description using AI, produces a prompt template or Python code, and saves the check. It explicitly distinguishes from sibling create_or_update_check by contrasting when to use each.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It provides explicit usage guidance: use this tool when only a description exists, and use create_or_update_check directly when the exact prompt/code is known. It also suggests refining the generated output with create_or_update_check, giving clear when-to and when-not-to context.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_checkGet CheckA
Read-onlyIdempotent
Inspect

Retrieve the full configuration of a check by name, including its prompt template or code contents.

    Args:
        name: Name of the check to retrieve.
        version: Optional check version number to pin. Omit (or leave null)
            for the most recent version. The response always lists every
            available version under "available_versions".
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes
versionNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With readOnlyHint and idempotentHint already covering safety, the description adds valuable behavioral context: the response always includes 'available_versions' and that omitting version returns the most recent. It also discloses that the configuration includes prompt template or code contents, going beyond the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with the main purpose in one sentence, followed by a compact Args block. Every sentence adds necessary context without redundant filler or repetition of schema information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a get-by-name endpoint with two parameters and an output schema, the description covers the optional version behavior and guarantees that all versions are returned in the response. It does not need to describe return fields because an output schema exists, and annotations handle safety semantics.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Despite 0% schema description coverage, the Args section fully documents both parameters: name identifies the check, and version is optional with pinning behavior and a clear directive to omit for the latest version. This adds meaning well beyond the bare schema types and default.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with 'Retrieve the full configuration of a check by name,' which clearly states the verb (retrieve), resource (check configuration), and scope (by name). It further distinguishes from list_checks by specifying 'including its prompt template or code contents,' making the specific use case unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage when you need a full check configuration (as opposed to a list), but it does not explicitly name alternatives or state when not to use it. The version guidance is useful but does not address tool selection among siblings like list_checks or get_driver.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_conversation_transcriptGet Conversation TranscriptA
Read-onlyIdempotent
Inspect

Retrieve the full conversation transcript for a single data point.

    Use this after get_test_run_results to drill into a specific
    conversation. Provide either scenario_index (1-based, from the
    scores summary) or test_id (UUID) to identify the conversation.

    Returns the complete message transcript (model_input), final
    output (model_result), per-turn check scores (metric_value),
    and the scenario seed data.

    Args:
        test_run_id: The UUID of the test run.
        scenario_index: 1-based index of the conversation within
            the test run. Visible in get_test_run_results output.
        test_id: UUID of the specific data point. Alternative to
            scenario_index.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
test_idNo
test_run_idYes
scenario_indexNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and idempotentHint=true, so the tool's safety profile is known. The description adds value by enumerating the exact return contents (model_input, model_result, metric_value, scenario seed data) and noting it returns a 'complete' transcript. It does not contradict annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured and front-loaded: it opens with the main purpose, then gives usage context, then details the return values, and finally lists parameters in an Args block. Despite its length, every sentence serves a purpose and adds necessary detail that the schema omits.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simple retrieval nature, good annotations, and existing output schema, the description covers all necessary context: what it does, when to use it, how to select the target conversation, and what it returns. No additional information is needed for a competent agent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, but the description fully compensates by explaining all three parameters: test_run_id as UUID, scenario_index as a 1-based index visible in get_test_run_results, and test_id as an alternative UUID. This goes beyond the bare schema, which lacks descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action ('Retrieve the full conversation transcript') and a specific resource ('a single data point'). It clearly distinguishes itself from sibling tools like get_test_run_results by noting it is for drilling into a specific conversation after summary results.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly says 'Use this after get_test_run_results to drill into a specific conversation,' providing clear context. It also explains how to identify the conversation (scenario_index or test_id) but does not explicitly mention when not to use the tool or name alternative tools for other purposes.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_dashboardGet DashboardA
Read-onlyIdempotent
Inspect

Retrieve a dashboard's full configuration by name.

    Args:
        name: Name of the dashboard to retrieve.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, covering the safety profile. The description adds the 'full configuration' detail, indicating the complete payload is returned, but it does not discuss error scenarios, permissions, or other non-safety behaviors. With annotations present, this is adequate but not rich.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The main description is a single, front-loaded sentence that immediately conveys the purpose. The Args block is redundant with the schema but is brief and does not add significant clutter. Overall, it is concise and well-structured.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity (one parameter, read-only, idempotent) and the existence of an output schema, the description is largely sufficient. It clearly states what it retrieves and how. A mention of when to prefer this over list_dashboards would improve completeness, but the core information is present.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema provides no parameter description (0% coverage), so the description must compensate. It does so by clarifying that the 'name' parameter is the dashboard's name, but this is only a minimal restatement of the parameter's title. It adds no detail about format, uniqueness, or validation, which is acceptable for a simple getter but does not go beyond the obvious.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action (retrieve), the resource (a dashboard's full configuration), and the qualifier (by name). This distinguishes it from sibling tools like list_dashboards, which lists dashboards, and save_dashboard/delete_dashboard, which modify them.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives clear context for when to use the tool: retrieve a specific dashboard's full configuration by name. It does not explicitly mention alternatives or exclusions, but the purpose is specific enough that the usage scenario is unambiguous.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_docsQuery Okareo DocumentationA
Read-onlyIdempotent
Inspect

Query the Okareo documentation system for information about Okareo primitives and workflows.

    Use this tool when the agent or user needs to understand how Okareo
    concepts work — Scenarios, Checks, Targets, Drivers, Evaluations,
    and Simulations.

    Two modes are available:
    - 'conceptual': Detailed technical documentation for agent reasoning.
      Default top_k=5 (returns up to 5 documentation entries).
    - 'user_legible': Plain-language explanations for human users.
      Default top_k=3 (returns up to 3 documentation entries).

    If the Okareo documentation service is unavailable (e.g. air-gapped
    environment), the tool returns a helpful error suggesting get_templates
    as a fallback.

    Args:
        query: The question to ask the Okareo documentation system. Be
            specific — e.g., 'How do Checks and Evaluations work together?'
            or 'What is a Driver persona?'
        mode: Documentation mode — 'conceptual' or 'user_legible'.
        top_k: Number of documentation entries to return. Defaults to 5
            for conceptual mode, 3 for user_legible mode. Maximum 10.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
modeYes
queryYes
top_kNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description goes well beyond the annotations (readOnlyHint, openWorldHint, idempotentHint, destructiveHint) by disclosing mode-specific behaviors: 'conceptual' default top_k=5, 'user_legible' default top_k=3, maximum top_k=10, and a helpful error message suggesting get_templates as a fallback. These details describe the tool's dynamic behavior and error handling, which are not inferable from annotations alone.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured and front-loaded: first sentence states purpose, followed by usage guidance, mode details, error behavior, and parameter explanations. While longer than average, every sentence adds value—no redundancy. The use of bullet points and an Args section makes it scannable and easy to parse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite having multiple parameters and no schema descriptions, the description is complete for a read-only documentation query tool. It covers all parameter semantics, mode behavior, defaults, and fallback scenarios. The presence of an output schema means return-value details are not required, so the description is sufficient for an agent to correctly select and invoke the tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema provides only parameter names and types (query, mode, top_k, with top_k nullable and default null). The description compensates fully by explaining each parameter: query with concrete examples ('How do Checks and Evaluations work together?'), mode with its two possible values ('conceptual' or 'user_legible'), and top_k with its mode-specific defaults and maximum value (10). This is essential since the schema has no descriptions (0% coverage).

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Query the Okareo documentation system for information about Okareo primitives and workflows.' It uses a specific verb (query) and resource (documentation system), and explicitly lists the concepts covered (Scenarios, Checks, Targets, Drivers, Evaluations, Simulations). This also distinguishes it from sibling tools like get_templates, which is mentioned as a fallback.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit guidance is provided: 'Use this tool when the agent or user needs to understand how Okareo concepts work.' It also suggests a specific alternative (get_templates) in the event the documentation service is unavailable. The description clearly indicates when to use this tool and provides a fallback option, making it easy to choose among siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_driverGet DriverA
Read-onlyIdempotent
Inspect

Retrieve a driver persona you've already configured.

    Retrieves a Driver by name, returning its full configuration including the
    persona prompt.

    Args:
        name: Name of the driver to retrieve.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare this as read-only and idempotent. The description adds that the full configuration including persona prompt is returned, which is useful context. No mention of error behavior, but given the simple read-only nature, this is acceptable and adds some value beyond annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and front-loaded with the primary purpose. The additional sentence and Args section add value without redundancy or unnecessary detail.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple getter with one parameter, an output schema, and read-only annotations, the description covers the core behavior and return content. It doesn't discuss error handling, but the output schema and simplicity mitigate the gap, making the tool adequately complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has a single required string parameter 'name' with no description. The description's Args section restates that name is the driver name, providing minimal additional meaning. It doesn't add details like uniqueness or format, but for a simple identifier it suffices given the low schema coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool retrieves an existing driver persona by name, returning its full configuration including the persona prompt. This distinguishes it from sibling list_drivers, which lists drivers, and create_or_update_driver, which creates/updates.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for fetching a previously configured driver by name. It doesn't explicitly mention alternatives or when not to use it, but the context is clear enough: use when you need a specific driver's full config. It provides no exclusion rules but the intended use case is evident.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_generation_modelGet Generation ModelA
Read-onlyIdempotent
Inspect

Read detailed information about a registered generation model.

    Returns the generation model's target LLM configuration, tags, creation
    time, and any warnings (e.g., if the target LLM has been deprecated).

    Args:
        name: Name of the registered generation model.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false. The description adds useful behavioral context by listing exactly what is returned (target LLM config, tags, creation time, warnings) and explains what warnings may indicate (deprecation). No contradiction with annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and well-structured: an opening summary, a sentence on return values, and the Args breakdown. Every sentence provides necessary detail with no redundancy or filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple read operation with one parameter and an output schema, the description is complete. It covers the purpose, return content, and parameter semantics. No additional context is needed for correct invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema provides only a parameter name with no description (0% coverage). The description's Args section explicitly states 'name: Name of the registered generation model,' fully compensating for the schema's lack of detail. This adds clear meaning beyond the bare type.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's action ('Read detailed information about a registered generation model') with a specific verb and resource. It distinguishes from sibling tools like list_generation_models by emphasizing detailed single-item retrieval.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for when to use the tool (to read details of an existing generation model), but it does not explicitly mention alternatives or exclusion criteria. It does not say 'use list_generation_models to list all models' which would make it a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_reps_baselineGet REPS Baseline MaterialA
Read-onlyIdempotent
Inspect

Serve REPS agent-evaluation baseline material (scenario banks, drivers, checks, eval configs).

    REPS is Okareo's agent-evaluation workbench: per-pillar baseline
    material for evaluating AI agents across R-reasoning, E-execution,
    P-performance, and S-security, plus shared explore/ probes and a
    profile/ example. The material is published as tagged releases of
    the okareo-tools repo; this tool serves the latest release so reps
    skills need no local copy of the tree.

    Two modes:
    - Discovery (omit `path`): list what files exist in the served
      release — the full tree, or one area via `pillar`. File lists
      change between releases, so always discover before fetching.
    - Fetch (provide `path`): return one file's exact content as
      published in the release. Use paths verbatim from discovery,
      e.g. 'S-security/scenarios/verification-gate.jsonl'.

    Every response carries the release tag it was served from (e.g.
    'v0.5.1') — record it in evaluation reports as baseline
    provenance. `stale: true` means the last release check failed and
    the content may lag the newest release.

    Args:
        pillar: Optional discovery filter. One of: R-reasoning,
            E-execution, P-performance, S-security, explore, profile.
            Omit to list the entire baseline tree (which also includes
            shared material outside these areas).
        path: Optional file path (relative to the baseline tree, as
            returned by discovery). Provide to fetch that file's
            content; omit for discovery.
        version: Optional release tag. Currently only the served tag
            is available; any other value returns an error naming what
            IS available. Omit to accept the served release.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
pathNo
pillarNo
versionNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate read-only, idempotent, and non-destructive behavior. The description adds context about release provenance, staleness (`stale: true`), and the served-release-only version constraint, which are not inferable from annotations. It does not contradict annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is comprehensive but efficiently structured: overview, two modes, args, and response metadata. Each sentence adds distinct value, and the formatting makes it easy to parse. There is no redundancy or filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's dual-mode complexity, the description fully covers discovery and fetch behavior, parameter semantics, release provenance, and staleness warnings. The presence of an output schema means return-value details are not required in the description, and the description still notes key response fields.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 0% parameter descriptions, so the description fully compensates. It thoroughly explains `pillar` (discovery filter with enumerated values), `path` (fetch mode, use verbatim from discovery), and `version` (served release only, error otherwise), including an example path.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource: 'Serve REPS agent-evaluation baseline material' and defines REPS, clearly distinguishing it from sibling tools that manage individual scenarios or checks. It precisely states that the tool serves baseline files from tagged releases.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It explicitly explains two modes (discovery vs fetch), provides a rule to always discover before fetching, and describes when to use the `pillar` filter and `version` parameter. It also notes that any version other than the served tag returns an error naming available versions, giving clear expected behavior.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_scenarioGet ScenarioA
Read-onlyIdempotent
Inspect

Read a scenario's metadata and all data rows.

    Look up by name or scenario ID. Returns scenario details and all
    input/result data rows.

    Args:
        name: Name of the scenario to retrieve.
        scenario_id: ID of the scenario to retrieve. Takes precedence over name.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameNo
scenario_idNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The annotations already declare the tool safe (readOnlyHint=true, idempotentHint=true, destructiveHint=false). The description adds useful behavioral details: scenario_id takes precedence over name, and it returns all input/result data rows. It doesn't contradict annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is front-loaded with a one-sentence summary, followed by a structured Args block. There is slight redundancy between the first two sentences, but overall it's concise and well-organized.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple read-only tool, the description covers the lookup method, return content, and parameter precedence. The output schema handles return structure. However, it doesn't specify behavior when no arguments are provided or when a scenario is not found, which is a minor gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description's Args section is the only source of parameter meaning. It clearly explains both 'name' and 'scenario_id', including the precedence rule, fully compensating for the empty schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Read') and identifies the resource ('a scenario's metadata and all data rows'), which clearly distinguishes it from sibling tools like list_scenarios or get_check. The purpose is unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context: use this tool to read a scenario's metadata and data rows, with lookup by name or ID. It doesn't explicitly name alternatives or state when not to use it, but the read-only nature and specific resource make the use case clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_targetGet TargetA
Read-onlyIdempotent
Inspect

Check the current configuration of a test target.

    Retrieves a Target by name. Works for all target types (Generation,
    Custom Endpoint, and Voice).

    For **custom_endpoint** Targets, the response is a flat envelope whose
    keys mirror the kwargs accepted by `create_or_update_target`, so a
    copilot can read the result, swap in a new name + secrets, and feed it
    back to create to clone the Target. Fields the backend keeps secret
    (those listed in `sensitive_fields`) appear with the literal value
    `"***REDACTED***"` — these MUST be replaced with real values before
    calling `create_or_update_target`, which rejects payloads still
    containing the sentinel.

    The `max_parallel_requests` field on custom_endpoint Targets is the
    same setting the Okareo web UI labels "max concurrency".

    For generation and voice Targets, the response shape is unchanged
    (kept stable for existing callers).

    Args:
        name: Name of the target to retrieve.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes well beyond the readOnly/idempotent annotations by revealing specific behavioral details: sensitive fields appear as '***REDACTED***' and must be replaced before create_or_update_target, the max_parallel_requests field matches the UI label 'max concurrency', and the response shape remains stable for generation/voice targets. These are valuable insights not derivable from annotations alone.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with a clear opening statement, followed by detailed but relevant caveats and an Args section. Every sentence adds value and the length is justified by the multi-target-type nuances and redaction behavior.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers all target types, special handling for custom_endpoint, redacted fields, field naming mapping, and stability guarantees. With an output schema present, the description does not need to list return fields. It is highly complete for a read-only getter tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has no property description (0% coverage), so the description compensates by stating 'name: Name of the target to retrieve.' This adds straightforward meaning to the single required parameter. Though minimal, it is sufficient for a simple string parameter.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function with a specific verb+resource: 'Check the current configuration of a test target' and 'Retrieves a Target by name.' It distinguishes itself from sibling tools like list_targets by indicating single-target retrieval by name and explicitly mentions support for all target types.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies when to use the tool (when needing a single target's configuration) and provides context for different target types, but it does not explicitly contrast with list_targets or state exclusions. The intended use is clear enough from the phrasing and sibling tool names.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_templatesGet Okareo TemplatesA
Read-onlyIdempotent
Inspect

Retrieve prompt templates for common Okareo patterns.

    Returns starter templates for building Okareo test components. These
    templates are served as static content from the MCP — no network calls
    required. Always available, even in air-gapped environments.

    Available templates:
    - basic_scenario: Template for creating a basic Okareo test scenario
    - boolean_check_prompt: Template for a pass/fail (boolean) check prompt
    - score_check_prompt: Template for a scored check prompt
    - check_code: Template for a code-based check (Python function)
    - target_validate_check_prompt: Template for validating target output
    - driver_prompt: Template for a Driver persona prompt
    - driver_voice_extension_prompt: Template for voice interaction extensions
    - analysis_check_prompt: Template for an analysis check (qualitative feedback)

    Args:
        template_name: Template identifier to retrieve. Omit to get a
            lightweight listing of all available templates (names and
            descriptions only). Provide a template_name to get the full
            template content. Valid values: basic_scenario,
            boolean_check_prompt, score_check_prompt, check_code,
            target_validate_check_prompt, driver_prompt,
            driver_voice_extension_prompt, analysis_check_prompt.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
template_nameNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations (readOnlyHint, idempotentHint, destructiveHint), the description adds valuable behavioral context: templates are served as static content with no network calls, always available even in air-gapped environments. It also explains the differing behavior when the parameter is omitted vs provided, which is not captured by annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with an introductory sentence, a note on static availability, a bulleted list of templates, and an Args block. Despite being long, it is concise because every line adds necessary information. It is front-loaded with the primary purpose and does not contain fluff.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has a single optional parameter and an output schema. The description explains what the output will be in both modes (listing or full content) and lists all valid template names. Given the output schema exists, no further return-value detail is needed. The description is complete for the tool's simplicity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema provides zero description coverage for template_name (just a title and type). The tool description compensates fully by documenting the parameter, listing all valid values, and explaining the behavior when omitted. This is exemplary parameter documentation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: 'Retrieve prompt templates for common Okareo patterns.' It uses a specific verb (retrieve) and resource (prompt templates), and the list of available templates further distinguishes this tool from any sibling tool. No ambiguity about what the tool does.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear usage guidance: omit template_name to get a listing, or provide it to get full template content. It also explains the static, offline nature of the content. However, it does not explicitly compare to alternative tools or state when not to use it, though no sibling tool serves the same purpose.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_test_run_resultsGet Test Run ResultsA
Read-onlyIdempotent
Inspect

Load the results of a specific test run.

    Look up by test run ID (UUID) or by name (returns the most recent run
    matching that name). Returns aggregate metrics and per-row check scores.

    By default, conversation transcripts (model_input/model_result) are
    excluded to keep responses concise. Set include_transcripts=True to
    include full transcripts. Use get_conversation_transcript to inspect
    a single conversation's transcript without loading all of them.

    Supports pagination via limit and offset for large result sets.

    Args:
        test_run_id: The UUID of the test run. Takes precedence over name.
        name: The name of the test run. Returns the most recent match.
        include_transcripts: Include full model_input and model_result in
            each data point. Defaults to False (scores only). Set True for
            full conversation transcripts.
        limit: Maximum number of data points to return. 0 (default) returns
            all data points. Use with offset for pagination.
        offset: Number of data points to skip. Defaults to 0.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameNo
limitNo
offsetNo
test_run_idNo
include_transcriptsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already provide readOnly, idempotent, and non-destructive hints. The description adds valuable behavioral context beyond annotations: transcripts are excluded by default for conciseness, test_run_id takes precedence over name, and name returns the most recent match. It does not contradict annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with a summary paragraph and an Args list. It is somewhat lengthy but every sentence adds useful detail. The front-loaded purpose and logical flow make it easy to scan. Slight redundancy exists between the prose and Args list, but it aids clarity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The tool has an output schema, so return values are covered. The description explains lookup methods, alternative tool for transcripts, pagination, and default behavior. It is complete for a read-only query tool with 5 parameters; no crucial information is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 0% description coverage, so the description carries full burden. The Args section provides detailed semantics for all 5 parameters: precedence of test_run_id, behavior of name, effect of include_transcripts, and pagination semantics of limit/offset. This far exceeds what the schema provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool loads results of a specific test run, with specific lookup methods (by UUID or name) and what it returns (aggregate metrics and per-row check scores). It distinguishes itself from sibling tools like get_conversation_transcript and list_test_runs.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly tells when to use this tool vs alternatives: 'Use get_conversation_transcript to inspect a single conversation's transcript without loading all of them.' It also explains when to use include_transcripts and pagination, giving clear contextual guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_voice_integrationGet Voice IntegrationA
Read-onlyIdempotent
Inspect

Retrieve a voice provider integration by id, including its status.

    Args:
        integration_id: The integration's id (from list_voice_integrations).
    
ParametersJSON Schema
NameRequiredDescriptionDefault
integration_idYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is well covered. The description adds that the result includes 'status,' which is useful, but doesn't cover other behavioral aspects like pagination, error handling, or auth requirements. Given the strong annotation coverage, the additional context is sufficient but not rich.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is brief and front-loaded with the core purpose in the first sentence. The Args section is minimal and directly tied to the schema parameter. Every sentence earns its place without redundancy or fluff. This is an example of efficient, well-structured documentation.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple get-by-id tool with one parameter, strong annotations (read-only, idempotent), and an output schema, the description covers the necessary context: what to pass and that it returns status. The only missing element is explicit behavior for non-existent ids (e.g., not-found error), but given the tool's simplicity and existing structured metadata, this is adequate.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema provides only the parameter name and type (integration_id: string) with no description. The tool description compensates by explaining the parameter's origin and meaning: 'The integration's id (from list_voice_integrations).' This adds valuable semantic context beyond the schema, telling the agent where to obtain the id and implying it should reference a valid existing integration.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's action: 'Retrieve a voice provider integration by id, including its status.' It uses a specific verb (retrieve), identifies the resource (voice provider integration), and differentiates from siblings like list_voice_integrations by specifying lookup by id. This leaves no ambiguity about what the tool does.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear context for when to use the tool: 'by id' and the parameter hint 'from list_voice_integrations' implies a prerequisite workflow—first list integrations to obtain the id, then retrieve details. It doesn't explicitly name alternatives or exclusions, but the usage context is clear enough for the intended purpose.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_voice_webhook_urlGet Voice Webhook URLA
Read-onlyIdempotent
Inspect

Get the inbound webhook endpoint for a voice provider.

    Paste the returned URL into the provider's console so its call traffic
    reaches Okareo monitoring.

    Args:
        provider: Voice platform — one of: retell, twilio, vapi, elevenlabs.
        public_id: The integration's public_id (from connect_voice_integration
            or get_voice_integration). Required for retell and twilio.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
providerYes
public_idNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, openWorldHint, idempotentHint, and destructiveHint:false. The description adds useful context beyond these annotations, such as the purpose of the URL (reaching Okareo monitoring) and the conditional necessity of public_id for certain providers. It doesn't describe return format or error behavior, but annotations cover safety/read-only aspects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and well-structured: a clear purpose sentence, a usage directive, and a neatly formatted Args list. Every sentence adds value without unnecessary verbosity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's low complexity, an output schema is present, and annotations are rich, the description adequately covers purpose, usage, and parameter semantics. It leaves no critical gaps for an agent to invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has no descriptions (0% coverage), but the description's Args section comprehensively explains each parameter: provider lists valid enum values, and public_id is defined with its source and when it is required. This fully compensates for the schema's lack of detail.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Get') and resource ('inbound webhook endpoint for a voice provider'), clearly distinguishing it from sibling integration tools like connect_voice_integration or get_voice_integration. It also states the intended use of the returned URL, further clarifying its purpose.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description clearly explains when to use the tool (to obtain a webhook URL for a voice provider) and what to do with the result (paste it into the provider's console). It notes provider-specific requirements for public_id, which is important context. However, it does not explicitly mention when not to use this tool or name alternative tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

ingest_conversationsIngest ConversationsAInspect

Submit completed voice conversations to Okareo for monitoring.

    Each conversation's turns become evaluable data points and any
    configured monitors run their checks automatically. Use this to feed
    production voice traffic (Retell, Twilio, VAPI, ElevenLabs, or a custom
    source) into Okareo monitoring.

    Conversations are validated individually: valid ones are ingested and
    invalid ones are returned in a "rejected" list — the batch is not
    all-or-nothing.

    Args:
        conversations: List of conversation objects. Each MUST include a
            "call_id" and at least one of: "transcript" (a list of
            {role, content, timestamp_ms} turns), "audio"
            ({"type": "url"|"voice_file_id"|"inline_b64", ...}),
            "recording_url", or "recording_bytes_b64". Optional per
            conversation: "context_token", "metadata", "tags" (tags drive
            monitor/filter-group matching), "diarization", "first_turn".
            When both a transcript and audio are supplied, the transcript
            takes precedence.
        project_id: Okareo project ID. Defaults to the account's project.
        mut_id: Optional model-under-test ID. Omit for pure monitoring —
            data points are then matched to monitors by their tags only.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
mut_idNo
project_idNo
conversationsYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.8/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Even with annotations present (readOnlyHint=false, openWorldHint=true), the description adds substantial behavioral context: turns become evaluable data points, monitors run automatically, validation is per-conversation (not all-or-nothing), rejected list is returned, and transcript precedence when both transcript and audio are provided. This goes beyond what annotations reveal.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is structured effectively: a clear opening sentence, a usage paragraph, a validation behavior sentence, and a well-organized Args list. Every sentence carries essential information for correct invocation, and the length is justified by the complexity of the conversation object structure. No fluff or redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (nested conversation objects, multiple optional fields, integration with monitors, validation semantics) and the presence of an output schema, the description covers all necessary aspects: purpose, use cases, parameter details, validation behavior, and tag/mut_id matching. It leaves no significant ambiguity for an agent to invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, but the description fully compensates with a detailed Args section. It explains the conversations array's required fields (call_id, at least one of transcript/audio/recording_url/recording_bytes_b64), optional fields, and the exact semantics of project_id and mut_id, including the default and tagging behavior. This is exemplary compensation for missing schema descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource+purpose: 'Submit completed voice conversations to Okareo for monitoring.' It clearly distinguishes this from sibling tools like run_simulation and run_test by focusing on production voice traffic ingestion. The scope is unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit context: 'Use this to feed production voice traffic (Retell, Twilio, VAPI, ElevenLabs, or a custom source) into Okareo monitoring.' It also clarifies behavior like per-conversation validation and mut_id omission for pure monitoring. However, it does not explicitly name alternatives or when-not-to-use conditions, so it falls short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_available_llmsList Available LLMsA
Read-onlyIdempotent
Inspect

Browse available LLMs from the Okareo registry.

    Returns all LLMs that can be used when registering a generation model
    for testing. Each entry has a name, display name, and provider. Use a
    model_name from this list when calling register_generation_model.
    
ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, and non-destructive behavior. The description adds useful context by stating the data source (Okareo registry) and the shape of entries (name, display name, provider). This goes beyond the basic safety profile and explains how the output should be consumed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is three sentences, front-loaded with the core purpose, and every sentence earns its place. It avoids fluff and directly states what, why, and how to use the result. Excellent conciseness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter listing tool with strong annotations and an output schema, the description is complete. It covers the registry source, the purpose (registration), and the result fields. No additional behavioral or usage context is needed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so the input schema is trivial. Per rubric, a baseline of 4 is appropriate. The description adds no parameter details (none exist), but it does explain the output usage, which is the only semantically relevant information needed.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function: 'Browse available LLMs from the Okareo registry' and 'Returns all LLMs that can be used when registering a generation model.' It distinguishes this from sibling tools like list_generation_models by explicitly tying it to the registry and the register_generation_model workflow.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit guidance: 'Use a model_name from this list when calling register_generation_model.' This tells the agent exactly when to use this tool. It doesn't discuss when not to use it or contrast with alternatives, but the usage context is clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_checksList ChecksA
Read-onlyIdempotent
Inspect

List available quality checks, grouped by category.

    Returns checks (both built-in and custom) available in your Okareo
    account, organized into `checks_by_category` using the platform's
    `__category:<Category>` tags; checks with no category appear under
    `uncategorized`. Select checks from the category matching your task
    AND modality: voice-specific categories (e.g. voice/audio quality)
    apply to voice simulations, while checks outside voice-specific
    categories are generally useful for both chat and voice. A check
    carrying multiple categories appears under each of them.

    Each check has a name, description, and output_data_type.
    output_data_type uses the server vocabulary: "bool" is a pass/fail
    check and "int" is a scored check — these correspond to output_type
    "pass_fail" and "score" in create_or_update_check and generate_check.
    Use these check names with run_test to evaluate model quality.

    Args:
        limit: Maximum number of checks to return (default 20), applied to
            the total before grouping. Use 0 for no limit.
        all_versions: When false (default), returns only the latest version of
            each check. When true, returns the full version history of every
            check, each entry annotated with its version number.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
limitNo
all_versionsNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the read-only and idempotent annotations, the description reveals key behavioral details: grouping by `__category:<Category>` tags, handling of uncategorized checks, duplicate entries for multi-category checks, the meaning of output_data_type values ('bool', 'int') and their mapping to output_type, and version behavior controlled by all_versions. This adds significant value beyond the structured annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but well-structured with clear paragraphs and an Args section. The first sentence captures the core purpose. Each subsequent sentence adds necessary context about grouping, output semantics, and parameters. It is appropriately verbose for the tool's complexity.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, the description doesn't need to detail return fields, but it still explains the `checks_by_category` organization, uncategorized fallback, and output_data_type semantics. It also covers edge cases like multi-category checks and version history. The tool is fully specified for an agent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has no property descriptions, so the description carries the full burden. The Args section thoroughly explains `limit` (default, semantics of 0, application before grouping) and `all_versions` (latest vs full history, version annotation). This exceeds what the schema provides.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with 'List available quality checks, grouped by category', clearly identifying the action (list) and resource (quality checks). It distinguishes itself from sibling tools focused on creating, updating, or deleting checks.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear guidance on how to choose checks: 'Select checks from the category matching your task AND modality' and explains that voice-specific categories apply to voice simulations. It also directs the user to use check names with run_test, establishing the tool's role in the evaluation workflow. While it doesn't explicitly contrast with sibling tools, the context is sufficient.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_dashboardsList DashboardsA
Read-onlyIdempotent
Inspect

List the analytics dashboards in your Okareo project.

    Args:
        limit: Maximum number of dashboards to return (default 20). Use 0
            for no limit.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
limitNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.3/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false. The description adds useful behavioral context about the limit parameter, specifically that 0 means no limit, which is not in the schema and informs the agent of a special behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two sentences and front-loaded with the action. The Args block is compact and directly relevant. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple list tool with one parameter, the description covers purpose and parameter behavior. The output schema covers return structure, so no need to describe it. It could mention pagination or ordering, but that isn't required for straightforward listing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema only provides the parameter type and default (20), while the description explains that limit is the maximum number of dashboards to return and that 0 disables the limit. This adds meaningful semantics beyond the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'List' and the resource 'analytics dashboards in your Okareo project.' It distinguishes from siblings like get_dashboard (which fetches a single dashboard) and other list_* tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The usage is implied by the name and description: use to list dashboards. However, there is no explicit guidance on when to prefer this over get_dashboard or other list tools, nor any mention of alternatives.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_driversList DriversA
Read-onlyIdempotent
Inspect

See what driver personas are available in this project.

Returns all Drivers with their names, IDs, model, and temperature.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate a safe read-only, idempotent operation. The description adds value by disclosing exactly what is returned (names, IDs, model, temperature) and that it returns all drivers, which goes beyond the minimal annotation information.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is two short sentences, front-loaded with 'See what driver personas are available' and immediately followed by the return fields. No wasted words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple list tool with no parameters, an output schema, and a clear description of the returned data, this is fully complete. The output schema handles detailed return shape, and the description provides the semantic content.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so there is nothing to explain. The description correctly avoids any parameter details, and the baseline for zero-parameter tools is 4.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool lists all driver personas with specific fields (names, IDs, model, temperature). This distinguishes it from related tools like get_driver (single driver) and list_driver_voices (voices only).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The context is clear: use this when you need an overview of all available drivers. It does not explicitly mention alternatives or exclusions, but the simplicity and scope ('all Drivers') make the use case obvious without extra guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_driver_voicesList Driver VoicesA
Read-onlyIdempotent
Inspect

Discover the voices, voice profiles, and languages available for configuring voice-capable drivers.

    Call this before create_or_update_driver when building a voice agent
    simulation, so you can pass valid `voice`, `voice_profile`, and
    `language` values.

    Each entry in `voices` carries selection metadata — use it to pick
    the voice:
    - `language`: bare ISO code (e.g. "en", "es", "ja"). The driver's
      `language` is derived from the selected voice's language when
      omitted, and must not conflict with it.
    - `accent`: free-text accent label present on many voices (e.g.
      "British", "Mexican", "Parisian", "Southern US"). To satisfy an
      accent request, select a voice whose `accent` matches — writing
      accent instructions into `voice_instructions` does NOT change the
      TTS voice.
    - `gender`: e.g. "feminine" / "masculine".

    `voice_profiles` are emotion/delivery presets (happy, angry,
    sarcastic, ...) — they shape affect, not accent or language.
    `languages` lists the distinct voice languages available.
    
ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, and non-destructive, so the description need not repeat these. It adds valuable context about the data semantics (e.g., `language` derivation and conflict rule, `accent` matching requirement) and clarifies the limitation of `voice_instructions`, going beyond the structured annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is longer than typical, but it earns its length with structured bullet points and front-loaded purpose/usage statements. Every sentence adds meaningful guidance, and the format aids readability despite the density.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has no parameters and an output schema exists, the description is fully complete for an agent to invoke it correctly. It covers the key concepts (voices, profiles, languages), selection criteria, and integration with create_or_update_driver, leaving no critical gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so the baseline is 4. The description compensates by thoroughly explaining the output fields (language, accent, gender) and their intended use, which is more relevant than parameter docs for this read-only list operation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource+scope: 'Discover the voices, voice profiles, and languages available for configuring voice-capable drivers.' It clearly distinguishes the tool from sibling list tools by focusing on voice configuration and explicitly references create_or_update_driver, making its purpose unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit when-to-use guidance: 'Call this before create_or_update_driver when building a voice agent simulation, so you can pass valid `voice`, `voice_profile`, and `language` values.' It also offers selection guidelines and a direct exclusion ('writing accent instructions into `voice_instructions` does NOT change the TTS voice'), effectively preventing misuse.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_generation_modelsList Generation ModelsA
Read-onlyIdempotent
Inspect

Browse all registered generation models in the project.

Returns generation model names, IDs, target LLM configurations, and creation timestamps. Use this to see what generation models are available for testing.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, establishing a safe read operation. The description adds scope ('registered', 'project') and return fields, providing modest extra context beyond annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is three short sentences, front-loaded with the action and resource. Every sentence contributes value: the action, the returned data, and the intended use case.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no parameters and an output schema present, the description sufficiently covers the purpose and return content. The mention of 'registered models in the project' clarifies scope, and the use case adds practical context.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, and schema coverage is trivially 100%. The description does not need to explain parameters; the baseline for 0 params is 4, and no additional param information is required.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool lists all registered generation models, specifying returned fields (names, IDs, target LLM configurations, creation timestamps). This distinguishes it from siblings like get_generation_model (single model) and list_available_llms (available LLMs).

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides a clear use case: 'Use this to see what generation models are available for testing.' It does not explicitly mention alternatives or when not to use this tool, but the context is sufficient.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_scenariosList ScenariosA
Read-onlyIdempotent
Inspect

List scenarios in the project, most recent first.

    Returns scenario names, IDs, tags, row counts, and creation dates.
    Use this to discover existing scenarios before running a test.

    Args:
        limit: Maximum number of scenarios to return (default 20).
            Set to 0 to return all scenarios.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
limitNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare this as read-only, idempotent, and non-destructive. The description adds meaningful behavioral context beyond annotations, such as 'most recent first' ordering and the semantics of the limit parameter (default 20, 0 returns all). No contradictions exist.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and well-structured: main behavior and ordering in the first sentence, returned fields in the second, usage guidance in the third, and a clean Args section. Every sentence adds value, and the format is easy to parse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

The description fully covers the single optional parameter and the tool's purpose. With an output schema present, it doesn't need to detail return structure. It could mention the project context more explicitly, but the description is sufficiently complete for a list tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema only provides type and default for the limit parameter, but the description fully explains its meaning: 'Maximum number of scenarios to return (default 20). Set to 0 to return all scenarios.' This goes well beyond the schema's minimal coverage.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool lists scenarios in the project, most recent first, and enumerates the returned fields (names, IDs, tags, row counts, creation dates). This specific verb+resource combination distinguishes it from siblings like get_scenario or save_scenario.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It provides a clear context: 'Use this to discover existing scenarios before running a test.' While it doesn't explicitly mention alternatives or when-not-to-use, the guidance is sufficient for a list operation, and no other sibling offers the same discovery functionality.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_simulationsList SimulationsA
Read-onlyIdempotent
Inspect

List past simulation runs in the project.

    Returns simulation run names, IDs, timestamps, and status, sorted by
    most recent first. Defaults to the 10 most recent runs in summary mode.

    Use detail_level="detailed" to include model_metrics and additional
    fields (limit is capped to 5 in detailed mode to prevent overflow).

    Use get_test_run_results with the returned test_run_id to retrieve
    per-row scores (transcripts excluded by default). Then use
    get_conversation_transcript with a scenario_index to inspect
    individual conversation transcripts.

    Args:
        target_name: Optional filter — only show simulation runs using
            this target.
        scenario_name: Optional filter — only show simulation runs using
            this scenario.
        limit: Maximum number of runs to return, sorted by most recent
            first. Defaults to 10. Set to 0 to return all runs.
        detail_level: "summary" (default) returns compact results without
            model_metrics. "detailed" returns full results with metrics
            (limit capped to 5).
    
ParametersJSON Schema
NameRequiredDescriptionDefault
limitNo
target_nameNo
detail_levelNosummary
scenario_nameNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description adds behavioral details beyond annotations, including sorting order ('sorted by most recent first'), default limit of 10, the cap at 5 for detailed mode, and the distinction between summary and detailed outputs. These are not conveyed by the readOnly/idempotent annotations, making the description valuable.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is well-structured with a clear summary, a usage workflow paragraph, and a labeled Args section. It is relatively detailed but each sentence serves a purpose, and the use of a code block for parameters improves readability.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the presence of an output schema (which covers return structure), the description covers the tool's purpose, all parameters, behavioral constraints, and its place in the workflow with sibling tools. It is complete enough for an agent to select and invoke the tool correctly without additional information.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has no parameter descriptions (0% coverage), but the description thoroughly explains each parameter: target_name and scenario_name as optional filters, limit with its default and 0-to-return-all behavior, and detail_level with its two allowed values and the 5-cap implication. This fully compensates for the schema's lack of documentation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's function with a specific verb and resource ('List past simulation runs in the project'), and details what is returned (names, IDs, timestamps, status, sorted most recent first). It differentiates from sibling tools like get_test_run_results and get_conversation_transcript by outlining the intended workflow.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It provides explicit guidance on when to use this tool versus downstream tools, instructing to use get_test_run_results with the returned test_run_id and get_conversation_transcript with scenario_index. It also explains when to switch detail_level based on whether model_metrics are needed. This contextualizes usage within the broader API workflow.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_targetsList TargetsA
Read-onlyIdempotent
Inspect

Browse all simulation targets available in this project.

    Returns all simulation targets (voice and custom_endpoint types)
    created via create_or_update_target. Does not include generation models
    registered via register_generation_model — use list_generation_models
    for those.
    
ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover read-only and non-destructive nature. Description adds that it returns only voice and custom_endpoint types created via create_or_update_target, and excludes generation models—useful scope context that complements the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Description is short, front-loaded with the main purpose, and every sentence provides value without redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple list operation with an output schema, the description covers the essential scope, exclusions, and fallback for related entities. No gaps identified.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

No parameters, so schema is empty. Description adds meaning by clarifying the implicit selection criteria (types and origin), exceeding the baseline for zero-parameter tools.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Clearly states the tool lists all simulation targets, distinguishing from generation models by naming the alternative list_generation_models. The verb 'browse' is less specific than 'list' but resource and scope are explicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly instructs to use list_generation_models for generation models, providing a clear when-not-to-use and an alternative. Also scopes to the current project.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_tenantsList TenantsA
Read-onlyIdempotent
Inspect

List every Okareo organization you have access to in this MCP session.

    The currently-active organization is marked ``is_current: true``. The
    active organization is determined at sign-in (the token this session
    presents is already scoped to it). Only available on OAuth-authenticated
    sessions; on Bearer-API-key sessions returns
    ``tenant_selection_requires_oauth``.
    
ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/destructive hints, and the description adds meaningful context: the is_current marker and the OAuth-only constraint, going beyond annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three sentences, each providing distinct value: scope, active marker, and authentication requirement. No fluff, well organized.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the output schema exists and the tool has no parameters, the description fully covers behavior, including edge cases like non-OAuth sessions.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool has zero parameters, so the description need not explain semantics. The description implicitly confirms no inputs are needed.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb ('List') and resource ('every Okareo organization you have access to'), and distinguishes itself from siblings like switch_tenant by focusing on listing rather than switching.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states when the tool is available (OAuth-authenticated sessions) and what happens on Bearer-API-key sessions, giving clear usage context and an alternative behavior.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_test_runsList Test RunsA
Read-onlyIdempotent
Inspect

List past test runs in the project.

    Returns test run names, IDs, timestamps, status, and summary scores,
    sorted by most recent first. Defaults to the 10 most recent runs.
    Optionally filter by model name, scenario name, or type.

    For simulation runs (type MULTI_TURN), use get_test_run_results with the
    returned test_run_id to retrieve full conversation transcripts and per-turn
    check scores.

    Args:
        model_name: Optional filter — only show test runs using this model.
        scenario_name: Optional filter — only show test runs using this scenario.
        limit: Maximum number of runs to return, sorted by most recent first.
            Defaults to 10. Set to 0 to return all runs.
        simulation_only: When True, return only MULTI_TURN simulation runs.
            Useful for browsing past simulation results without NL_GENERATION or
            other test run types appearing in the list.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
limitNo
model_nameNo
scenario_nameNo
simulation_onlyNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, non-destructive; description adds default limit=10, limit=0 returns all, and that results are sorted by most recent. Also clarifies that simulation_only filters to MULTI_TURN type.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is organized with a clear opening sentence, followed by return values, defaults, filters, and a pointer to an alternative tool. The Args section is structured and each line adds value. No wordiness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With a rich output schema present, the description need not detail return types, but it already summarizes return fields. It covers filters, defaults, and the recommended next step for transcripts. Sufficient for an agent to use correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema has zero field descriptions, but the description documents every parameter (model_name, scenario_name, limit, simulation_only) with defaults and purpose, fully compensating. The simulation_only explanation gives extra context on what MULTI_TURN simulation runs are.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States it lists past test runs, specifies returned fields (names, IDs, timestamps, status, summary scores) and sort order. Clearly differentiates from get_test_run_results by directing users there for transcripts.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly directs users to get_test_run_results for simulation run details, implying this tool is for summary listing. Explains filtering options with examples of when simulation_only is useful.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_voice_integrationsList Voice IntegrationsA
Read-onlyIdempotent
Inspect

List the voice provider integrations in your Okareo project.

    Args:
        limit: Maximum number of integrations to return (default 20). Use 0
            for no limit.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
limitNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, openWorldHint=true, idempotentHint=true, destructiveHint=false, so the safety profile is covered. Description adds the limit semantics (default 20, 0 for no limit), which is useful but does not disclose other behaviors like pagination or ordering.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Description is short, front-loaded with a clear sentence, and an organized Args section. No redundant words.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-parameter, read-only list tool with an output schema, the description covers the core purpose and parameter semantics. The scoped context and limit behavior are sufficient.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The description explicitly explains the limit parameter, including the default value and the special meaning of 0 (no limit), which goes beyond the schema's default. This compensates for the low schema coverage, even though there is only one parameter.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states the action (list), resource (voice provider integrations), and scope (in Okareo project), distinguishing it from sibling tools such as get_voice_integration and connect_voice_integration.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use versus alternatives, such as get_voice_integration for a single integration. Usage is implied by the list operation, but no exclusions or use-case contexts are provided.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

preview_delete_scenarioPreview Scenario DeletionA
Read-onlyIdempotent
Inspect

Preview what will be deleted before removing a scenario.

    Shows the scenario details and count of related test runs that will
    also be deleted. Use delete_scenario to confirm deletion after reviewing.

    Args:
        name: Name of the scenario to preview deletion for.
        scenario_id: ID of the scenario. Takes precedence over name.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameNo
scenario_idNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is covered. The description adds valuable behavioral context beyond the annotations: it discloses that the tool shows scenario details and the count of related test runs, and that scenario_id takes precedence over name. This goes beyond what annotations provide, though it could mention edge cases like missing both parameters.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and well-structured, with the main purpose stated in the first sentence and an explicitly labeled Args block. Every sentence contributes meaningful information without fluff or redundancy. The front-loaded design aids quick comprehension.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's simplicity, the presence of an output schema (which covers return values), and the strong annotations, the description is complete for an agent to select and invoke the tool correctly. It covers the tool's purpose, the follow-up action, and the parameter nuances, making it fully adequate for the task.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema provides no descriptions for the two optional parameters (0% coverage), so the description carries this burden. It explains the meaning of 'name' and 'scenario_id' and the precedence relationship, which is essential for correct usage. However, it does not explicitly state that at least one parameter should be provided, leaving a minor ambiguity.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Preview what will be deleted before removing a scenario.' It specifies the action (preview) and the resource (scenario deletion), and it distinguishes itself from the sibling tool delete_scenario by explicitly naming the confirmation workflow. It also details what will be shown: scenario details and the count of related test runs.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit usage guidance: 'Use delete_scenario to confirm deletion after reviewing.' This tells the agent when to use this tool versus the alternative, making the workflow clear. The contextual hint that this is a pre-deletion review step is sufficient for tool selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

query_analyticsQuery AnalyticsA
Read-onlyIdempotent
Inspect

Query Okareo's product analytics to understand evaluation trends.

    Answers questions like "how is my evaluation quality trending" by
    aggregating measures across dimensions over a time window.

    Args:
        measures: Metrics to aggregate. Required. For the ``check_trend``
            cube: avg_check_value, issue_rate, error_rate, datapoint_count,
            issue_count, error_count, test_run_count, avg_latency, sum_cost,
            input_token_count, output_token_count.
        dimensions: Optional group-by fields (e.g. ["check.name"],
            ["target.name"], ["provider"]).
        cube: Optional analytics cube name (defaults to ``check_trend``,
            currently the only cube).
        filters: Optional list of filter objects
            ``{"member": ..., "operator": ..., "values": [...]}``.
        time_range: Optional look-back window — one of LAST_HOUR,
            LAST_24_HOURS, LAST_7_DAYS, LAST_14_DAYS, LAST_30_DAYS,
            LAST_90_DAYS. If neither time_range nor time_dimensions is
            given, defaults to LAST_30_DAYS (the analytics API requires a
            time window).
        time_dimensions: Optional time bucketing — a list with at most one
            entry, e.g. [{"dimension": "test_run.start_time",
            "granularity": "day"}] (granularity: hour, day, or week).
        include_metadata: When true, also return the available cubes,
            dimensions, and measures so the query can be refined.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
cubeNo
filtersNo
measuresYes
dimensionsNo
time_rangeNo
time_dimensionsNo
include_metadataNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already mark it as read-only and idempotent. The description adds useful behavioral context: the API requires a time window and defaults to LAST_30_DAYS, and include_metadata returns available cubes/dimensions/measures. This goes beyond the annotations without contradicting them.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is lengthy but every sentence adds value, especially given the 7 parameters. The structured Args list improves scannability. Slight verbosity in the intro sentence could be trimmed, but it's well-organized.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a complex tool with 7 parameters and an output schema, the description covers all necessary context: defaults, constraints, allowed values, and metadata option. The output schema exists, so return-value details are not required. No gaps identified.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, but the Args section fully documents each parameter: valid measure names, dimension examples, cube options, filter object format, time_range enum, time_dimensions structure with granularity, and include_metadata behavior. This is exceptional compensation for the schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb and resource: 'Query Okareo's product analytics to understand evaluation trends.' It clearly differentiates from sibling tools like get_check or list_test_runs by focusing on aggregation and trend analysis. The example question reinforces the purpose.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives a concrete use case ('how is my evaluation quality trending') and notes the time-window requirement. It doesn't explicitly name alternative tools, but the context makes when to use this tool clear. Sibling names are mostly CRUD operations, so the differentiation is implicit.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

reevaluate_test_runRe-evaluate Test RunAInspect

Re-score a completed test run against a set of checks.

    Re-runs checks against an already-finished test run without re-executing
    the original model or simulation, and without changing the original
    run's results. Useful after a check definition changed, or to score an
    existing run against additional checks.

    Args:
        test_run_id: UUID or name of a completed test run.
        checks: Optional list of check names (or IDs) to score against.
            When omitted, the run's existing checks are re-run.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
checksNo
test_run_idYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.6/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Discloses that it does not re-execute the original simulation and does not change original run results, adding valuable context beyond the annotations. It does not mention idempotency or other side effects, but the output schema covers return behavior, and the annotation openWorldHint already signals possible side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise, front-loaded with the main purpose, and includes a structured Args list. Every sentence adds value, making it easy to scan and understand quickly.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, the description does not need to explain return values. Prerequisites, use cases, and parameter semantics are all clearly covered, so the tool can be invoked correctly without further information.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema has zero parameter descriptions, yet the description fully explains both parameters: test_run_id is a UUID/name of a completed run, and checks is an optional list with clear behavior when omitted. This completely compensates for the schema gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Description clearly states it re-scores a completed test run without re-executing the model/simulation, distinguishing it from run_test and get_test_run_results. The verb 're-score' plus the qualifier 'without re-executing' provides precise scope and uniqueness.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly notes it is useful after a check definition changed or to score against additional checks, giving clear when-to-use context. However, it does not explicitly name alternative tools or state when not to use this tool, so it stops short of full guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

register_generation_modelRegister Generation ModelA
Idempotent
Inspect

Register a generation model for testing by selecting an LLM from the registry.

    Creates a generation model (Model Under Test) that points to a specific LLM
    (e.g., 'azure/gpt-4o-mini'). Use list_available_llms to see available
    LLMs. The registered generation model can then be used with run_test.

    Args:
        name: A human-readable name for this generation model (e.g., 'my-chatbot').
        model_name: The LLM from the registry (from list_available_llms).
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes
model_nameYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations indicate idempotentHint=true, which is helpful, and the description adds that this 'creates a generation model (Model Under Test)' pointing to a specific LLM. It clarifies the registration semantics (registering creates an instance) and the relationship to testing, which goes beyond the annotations. It doesn't mention persistence or side effects, but the idempotentHint covers re-registration safety.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and well-structured. It front-loads the core purpose, provides usage guidance, and ends with parameter explanations. Every sentence earns its place, and the formatting with Args clearly separates parameter descriptions.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's moderate complexity, the description fully covers what an agent needs: the purpose, the required parameters, how to discover valid model_name values, and how the result is used. The presence of an output schema means the description doesn't need to explain return values. The sibling context (list_available_llms, run_test) is addressed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description carries the full burden. It explains 'name' as 'a human-readable name' and 'model_name' as 'the LLM from the registry (from list_available_llms)', which adds meaning beyond the raw schema types. It could provide more detail on name uniqueness, but the description effectively maps parameters to their semantics.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's purpose: 'Register a generation model for testing by selecting an LLM from the registry.' It uses specific verbs ('register', 'creates') and identifies the resource ('generation model') and distinguishes it from sibling tools like list_generation_models or update_generation_model. The example LLM ('azure/gpt-4o-mini') further clarifies the intended use.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly tells users to use 'list_available_llms' to see available LLMs and mentions that the registered model can be used with 'run_test'. This provides clear when-to-use context and names alternatives/prerequisites, which is excellent guidance for an AI agent.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

reorder_dashboardsReorder DashboardsB
Idempotent
Inspect

Set the display order of dashboards.

    Args:
        ordered_names: Dashboard names in the desired order.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
ordered_namesYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

B3.4/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already disclose readOnlyHint=false, destructiveHint=false, and idempotentHint=true, so the description doesn't need to repeat those. However, it adds no additional behavioral context, such as whether the order is fully replaced or how unlisted dashboards are handled. The description provides no new behavioral insight beyond the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise: a single sentence stating the purpose followed by a brief parameter definition. It is front-loaded and avoids unnecessary wording. The only minor issue is the embedded 'Args:' block format, but it remains efficient and easy to parse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple reorder operation with one parameter and an output schema, the description provides the core information needed. However, it omits edge-case behavior, such as whether all dashboards must be included or how partial lists are handled. The tool is straightforward, but the description could be more complete with a note on ordering semantics.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With schema description coverage at 0%, the description compensates by explicitly defining the parameter: 'ordered_names: Dashboard names in the desired order.' This adds meaningful context beyond the schema's bare 'array of strings' type. It clearly explains what the parameter represents and how to use it.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool's action: 'Set the display order of dashboards.' This uses a specific verb and resource, distinguishing it from sibling tools like list_dashboards, save_dashboard, or delete_dashboard. The purpose is immediately clear and unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides no explicit guidance on when to use this tool versus alternatives. It only states the action without explaining prerequisites (e.g., dashboards must exist) or how it relates to other dashboard operations. The usage context is heavily implied rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

rotate_voice_integration_secretRotate Voice Integration SecretA
Destructive
Inspect

Rotate a voice provider integration's secrets.

    Args:
        integration_id: The integration's id.
        secrets: The new provider secret values. The response returns only
            a secret summary, never raw secret values.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
secretsYes
integration_idYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate destructive and non-read-only behavior. The description adds valuable context by stating 'The response returns only a secret summary, never raw secret values,' which goes beyond the annotations and informs the agent about the response behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise: one main sentence plus an inline parameter list. Every word earns its place, and the main action is front-loaded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has only two parameters, a nested object, and an output schema, the description covers the essential intent and the notable response behavior. It could mention prerequisites or side effects of rotation, but the existing annotations and the explicit note on response summaries make it fairly complete.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema description coverage, the description provides basic meaning for both parameters: integration_id is 'the integration's id' and secrets are 'the new provider secret values.' This is helpful but lacks detail on format, required structure, or constraints for the nested secrets object.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action: 'Rotate a voice provider integration's secrets.' The verb 'rotate' is specific and distinct from sibling tools like connect, update, get, or delete, making the tool's purpose unambiguous.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for rotating secrets but gives no explicit guidance on when to use this tool versus alternatives like update_voice_integration. There is no mention of exclusions or when-not-to-use conditions, though the specific action is self-evident.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_simulationRun SimulationAInspect

Run a multi-turn conversation evaluation of your AI agent.

    Combines a Target (the system under test), a Driver (the simulated user),
    and a Scenario (the test cases) to generate realistic multi-turn conversations
    and evaluate them with quality checks.

    Returns promptly so the call never times out on long runs. Short runs that
    finish within the buffer window return ``status: "finished"`` with results
    ready; longer runs return ``status: "running"`` with the ``test_run_id``,
    ``app_link``, and an ``estimated_runtime`` — the run continues to completion
    on its own. In both cases, poll get_test_run_results with the returned
    test_run_id for scores, and get_conversation_transcript for transcripts.

    To rerun a previous simulation — keeping its configuration but changing one or
    more parameters — pass based_on_run_id with the original run's ID and supply
    only the values you want to override. If scenario_name or target_name are
    omitted and based_on_run_id is provided, they will be resolved from the
    original run.

    For custom_endpoint Targets: an exception raised during the run (for
    example the endpoint erroring mid-conversation) FAILS the run — it is
    reported as a failed simulation, not silently skipped.

    **Voice augmentations** — for voice Targets, the `augmentation` parameter
    applies realistic acoustic and conversational effects. Six top-level keys:
    `cap`, `directed_speech`, `secondary_speaker`, `backchannel`, `barge_in`,
    plus the composable `noise`. **Composition rule**: at most one non-noise
    strategy may be active, optionally combined with `noise`. Augmentations
    apply only to voice Targets — calls against generation or custom_endpoint
    Targets with an augmentation block are rejected. Field-level errors
    (out-of-range probability, missing required field, swapped offsets, unknown
    strategy) are returned by the MCP before any backend call.

    Strategy required / optional fields (numeric ranges in brackets):
      - cap: probability [0.0, 1.0] required. pause_ms [0, 10000] optional.
      - directed_speech: probability [0.0, 1.0] required. lpf_cutoff_hz (>0),
        gain_db [-40.0, 0.0], sample_rate (>0), prompt, reverb_preset optional.
      - secondary_speaker: probability [0.0, 1.0] AND secondary_voice (non-empty
        string) required. inter_speaker_pause_ms [0, 5000], lpf_cutoff_hz (>0),
        gain_db [-40.0, 0.0], sample_rate (>0), secondary_prompt,
        secondary_voice_instructions, secondary_reverb_preset optional.
      - backchannel: utterance (non-empty string) required. probability
        [0.0, 1.0], min_offset_ms (>=0), max_offset_ms (>= min_offset_ms),
        seed optional.
      - barge_in: prompt (non-empty string) required. probability [0.0, 1.0],
        min_offset_ms (>=0), max_offset_ms (>= min_offset_ms), seed optional.
      - noise: noise_profile (non-empty string) AND noise_snr_db (number)
        required. seed optional.

    For copy-paste examples and the full reference, call
    `get_templates(["voice_augmentations"])`.

    Args:
        name: Human-readable name for this simulation run.
        scenario_name: Name of the scenario to use. Required unless based_on_run_id
            is provided and the original run's scenario can be resolved.
        target_name: Name of the target to evaluate. Required unless based_on_run_id
            is provided and the original run's target can be resolved.
        driver_name: Name of the driver persona. If omitted, the project default
            driver is used.
        checks: List of check names to apply (from list_checks). Pick from
            the list_checks category matching the task and modality —
            voice-specific categories for voice simulations, categories
            outside them for either modality; never chat-only checks for
            audio (or vice versa). Every simulation runs with at least one
            check: when omitted or empty, the benign code-based "latency"
            performance check is applied automatically and the response
            discloses the substitution via `default_check_applied`.
            Supplied checks are used unchanged.
        repeats: Number of times to run each scenario row, default 1.
        max_turns: Maximum conversation turns per simulation, default 5.
        first_turn: Who speaks first — 'target' or 'driver', default 'target'.
        based_on_run_id: ID of a previous simulation run to reuse parameters from.
            Explicitly supplied values override the original run's parameters.
        augmentation: (voice Targets only) Voice augmentation block. See the
            "Voice augmentations" section above for keys and ranges. An empty
            dict is treated as no augmentation.
        turn_transition_time: Milliseconds of pause between turns. Forwarded to
            the backend as-is; SDK default (1000) is used when omitted.
        silence_timeout_ms: The target reply timeout — how patient Okareo
            is before indicating that the target can't respond. Do NOT set
            or change this value unless the user specifically directs it;
            it should be 10000 ms in nearly all cases. It exists to
            accommodate untuned targets with very long tool calls, during
            which the Driver waits patiently. It does NOT change how fast
            Okareo responds, and lowering it does not speed anything up —
            a slow simulation is not a reason to change it. Forwarded to
            the backend; backend default is used when omitted.
        checks_at_every_turn: When True, checks are evaluated per turn (not
            only at end of run).
        stop_check: Early-stop config: `{"check_name": str, "stop_on": <value>}`.
            The run halts as soon as the named check returns `stop_on`.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes
checksNo
repeatsNo
max_turnsNo
first_turnNotarget
stop_checkNo
driver_nameNo
target_nameNo
augmentationNo
scenario_nameNo
based_on_run_idNo
silence_timeout_msNo
checks_at_every_turnNo
turn_transition_timeNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations indicate readOnlyHint=false and openWorldHint=true, and the description adds substantial behavioral context: asynchronous execution ('Returns promptly so the call never times out... run continues to completion on its own'), status values ('finished' vs 'running'), and the failure mode for custom_endpoint targets ('exception raised during the run FAILS the run'). It also discloses default check substitution and field-level validation before backend calls, with no contradiction to annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description front-loads a clear one-sentence summary and is well-structured with sections and an Args list, making it easy to scan. However, it is quite long, and the voice augmentation section provides field-level detail that overlaps with the instruction to call get_templates for the full reference, introducing some redundancy that could be trimmed.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity (14 parameters, async behavior, voice augmentation constraints), the description is remarkably complete. It covers all parameters, return statuses, the polling workflow, failure modes, and check handling. The presence of an output schema means return values need not be repeated, and the description provides all necessary context for successful invocation.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description carries the full burden, and it does thoroughly. The Args section explains every parameter beyond schema types, e.g., 'repeats: Number of times to run each scenario row', 'silence_timeout_ms: Do NOT set... should be 10000 ms', and the augmentation block is fully detailed with required/optional fields and numeric ranges. This far exceeds minimal compensation for the schema gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with 'Run a multi-turn conversation evaluation of your AI agent,' a specific verb+resource+modality that clearly states the tool's purpose. It distinguishes from siblings by emphasizing multi-turn conversation evaluation and pointing to follow-up tools like get_test_run_results and get_conversation_transcript, setting it apart from run_test and other simulation-related tools.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides explicit when-to-use and when-not-to-use guidance: silence_timeout_ms should not be changed unless requested, augmentation is rejected for non-voice targets, and checks are sourced from list_checks. It also directs users to get_templates for examples and get_test_run_results for polling, clearly naming alternatives and context for use.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

run_testRun TestAInspect

Submit a quality test that evaluates a model against a scenario using checks.

    Returns promptly so the call never times out on long runs. Short runs return
    ``status: "finished"`` with results ready; longer runs return
    ``status: "running"`` with the ``test_run_id`` and ``app_link`` — the run
    continues to completion on its own. In both cases, poll get_test_run_results
    with the returned test_run_id to retrieve scores.

    Args:
        scenario_name: Name of the scenario to evaluate against.
        model_name: Name of the registered model to evaluate.
        checks: List of check names to apply (e.g., ["coherence", "relevance"]).
            Use list_checks to discover available checks and pick from the
            category matching the task and modality — do not use
            voice-specific checks for text evaluations (or vice versa);
            checks outside voice-specific categories suit both.
        name: Optional human-readable name for this test run.
        type: Type of evaluation. Defaults to NL_GENERATION. Valid values:
            NL_GENERATION, INFORMATION_RETRIEVAL, MULTI_CLASS_CLASSIFICATION,
            INVARIANT, MULTI_TURN, AGENT_EVAL.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameNo
typeNoNL_GENERATION
checksYes
model_nameYes
scenario_nameYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.9/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already mark this as non-read-only and non-idempotent, but the description adds essential async behavior: it returns promptly, may return status 'finished' or 'running' with test_run_id and app_link, and continues on its own. This fully discloses the lifecycle and interaction pattern.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact but dense, with no fluff. The Args block is a simple list within a paragraph, which could be more readable as a bulleted list, but every sentence earns its place and it avoids redundancy with the schema.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 5-parameter tool with an output schema present, the description covers all dimensions: what it does, how to use it, async behavior, parameter semantics, and interaction with sibling tools. The output schema handles return value details, so no additional return documentation is needed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema parameter descriptions are absent (0% coverage), so the description carries full responsibility. It defines every parameter: scenario_name, model_name, checks with an example and guidance on selection, name as optional, and type with all valid enum values. This compensates completely for the schema gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a specific verb+resource: 'Submit a quality test that evaluates a model against a scenario using checks.' This clearly distinguishes run_test from siblings like run_simulation or get_test_run_results.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explains when to use this tool for submitting a test, notes that short runs finish immediately while long runs return a status and ID, and explicitly directs polling via get_test_run_results. It also advises using list_checks and warns against mismatched check categories, providing actionable when-to-use guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

save_dashboardSave DashboardA
Idempotent
Inspect

Create or update an analytics dashboard by name (upsert).

    If a dashboard with this name already exists it is updated; otherwise a
    new one is created.

    Size each panel with a named ``size`` from the catalog below (PREFERRED
    — guarantees a legible layout) and omit positions entirely: panels are
    auto-placed in the order given (left-to-right, top-to-bottom on a
    12-column grid, wrapping rows, never overlapping).

    Size catalog and when to use each:

    - ``small-square`` (3x6): single ``stat`` metrics.
    - ``half-rectangle`` (6x6): ``line``/``bar``/``area`` trends, two per row.
    - ``half-square`` (6x9): ``radar``, ``composed``, denser charts.
    - ``full-rectangle`` (12x9): wide time-series comparisons.
    - ``full-square`` (12x12): ``table`` panels.

    Args:
        name: Dashboard name — the upsert key.
        panels: Optional list of panel definitions. Each panel is an object:

            - ``title`` (str, required): panel heading.
            - ``chart_type`` (str, required): one of ``line``, ``bar``,
              ``composed``, ``area``, ``radar``, ``stat``, ``table``.
            - ``query`` (object, required): what to chart —
              ``{"cube": "check_trend", "measures": [...],
              "dimensions": [...], "filters": [...],
              "time_dimensions": [...], "order": {...}}``. ``measures`` is
              required; everything else is optional. ``cube`` defaults to
              ``check_trend``. The dashboard ``time_range`` applies to all
              panels — panels do NOT carry their own time range.
            - ``size`` (str): a catalog name (see above). Required unless
              ``layout`` is given; wins over ``layout`` w/h if both appear.
            - ``layout`` (object): raw grid placement
              ``{"x": >=0, "y": >=0, "w": >=1, "h": >=1}`` (integers).
              Only needed when not using ``size``, or to pin an explicit
              position (give both ``x`` and ``y``; with ``size``, w/h are
              ignored). Heights below the legibility floor are sized up on
              save: ``h >= 6`` when ``w <= 6``, ``h >= 9`` when ``w > 6``.
            - ``table_config`` (object, optional): ONLY for
              ``chart_type == "table"``.

            ``check_trend`` measures: ``avg_check_value``, ``issue_rate``,
            ``error_rate``, ``datapoint_count``, ``issue_count``,
            ``error_count``, ``test_run_count``, ``avg_latency``,
            ``sum_cost``, ``input_token_count``, ``output_token_count``.
            ``check_trend`` dimensions: ``check.name``, ``check.id``,
            ``target.name``, ``target.id``, ``scenario.name``,
            ``scenario.id``, ``test_run.id``, ``test_run.type``,
            ``test_run.is_latest_for_target``, ``source``, ``provider``,
            ``request_model_name``, ``response_model_name``, ``tag``.
            Use ``query_analytics(include_metadata=True)`` for the
            authoritative, current set.

            Example panel::

                {"title": "Avg Check Value by Check", "chart_type": "bar",
                 "query": {"measures": ["avg_check_value"],
                           "dimensions": ["check.name"]},
                 "size": "half-rectangle"}
        description: Optional dashboard description.
        time_range: Optional default look-back window for the whole
            dashboard. One of: LAST_HOUR, LAST_24_HOURS, LAST_7_DAYS,
            LAST_14_DAYS, LAST_30_DAYS, LAST_90_DAYS. Defaults to
            LAST_90_DAYS when omitted.

    Returns:
        JSON with the saved dashboard and ``action`` (created/updated).
        When sizing or dimensions were changed on save (size overriding
        layout w/h, or a height floored), an ``adjustments`` list reports
        each change: ``{"panel", "field", "from", "to", "reason"}``.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes
panelsNo
time_rangeNo
descriptionNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.7/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations (idempotentHint=true, destructiveHint=false), the description details many behavioral traits: upsert behavior, auto-placement of panels, size overriding layout w/h, height flooring adjustments, and the return of an 'adjustments' list. This adds substantial context beyond what annotations provide.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is long but well-structured with clear sections, bullet points, and an example. Every sentence adds value, though it could be slightly tightened. The front-loaded first sentence states the core purpose immediately.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool's complexity and the sparse schema, the description is exhaustive: it explains return values (including 'action' and 'adjustments'), defaults (time_range defaults to LAST_90_DAYS), the query object, and how to get authoritative dimensions/measures. It fully compensates for the lack of schema descriptions.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema is extremely sparse (0% coverage), so the description carries the full burden. It thoroughly explains each parameter: name, panels (with full panel object structure including title, chart_type, query, size, layout, table_config), time_range allowed values, and description. It also enumerates measures/dimensions and provides an example.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with 'Create or update an analytics dashboard by name (upsert)', which uses a specific verb ('create or update'), identifies the resource ('analytics dashboard'), and highlights the upsert behavior. This clearly distinguishes it from siblings like get_dashboard, list_dashboards, and delete_dashboard.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides clear guidance on when to use the tool (upsert semantics) and internal usage guidelines such as the size catalog and when to use size vs. layout. However, it does not explicitly contrast with sibling tools (e.g., 'use get_dashboard to view rather than save'), though the purpose statement implies it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

save_scenarioSave ScenarioA
Idempotent
Inspect

Save a named scenario for use in quality tests.

        Provide EXACTLY ONE dataset source: `content`, `file_path`, or
        `rows`.

        Prefer `file_path` for local .jsonl files — the server reads the
        file directly, so no rows pass through the assistant's context.
        When passing rows through the assistant instead, keep the dataset
        UNDER 2,000 rows (`content` with the file's text, or `rows` for
        tiny datasets). For 2,000 rows or more, always use `file_path` or
        upload directly to Okareo via the web app, SDK, or CLI, to avoid
        unnecessary token cost.

        If a scenario with the same name already exists, the existing
        scenario is returned (idempotent). Scenarios are immutable after
        creation — use create_scenario_version to create updated versions.

        Args:
            name: A unique name for the scenario.
            content: Raw JSONL text (one JSON object with 'input' and
                'result' per line). Only for datasets under 2,000 rows.
            file_path: Path to a local .jsonl file. Preferred — works for
                any size.
            rows: List of data rows, each with 'input' (any type) and
                'result' (any type). Use for small scenarios (< 20 rows).
            tags: Optional list of tags for categorizing the scenario.
        
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes
rowsNo
tagsNo
contentNo
file_pathNo

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A5/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Confirms idempotent behavior (existing scenario returned) and adds that scenarios are immutable after creation. Also discloses that file_path avoids passing rows through assistant context and notes token cost considerations, going beyond the idempotentHint annotation.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The structure is well-organized: opening purpose, usage guidance, idempotency note, and a clear Args list. Each sentence provides necessary information without unnecessary fluff, though slightly long due to parameter guidance.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Covers parameter selection, constraints, idempotency, immutability, and alternatives. Given that an output schema exists, it needn't describe return values; it's complete for a tool of this complexity.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters5/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With 0% schema coverage, the description provides detailed semantics for all 5 parameters, including constraints (e.g., 'one JSON object with input and result per line' for content, '< 20 rows' for rows) and trade-offs between sources.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states 'Save a named scenario for use in quality tests,' using a specific verb and resource. It distinguishes itself from sibling tools by explicitly noting that scenarios are immutable and that create_scenario_version is used for updates.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Provides explicit guidance on providing EXACTLY ONE dataset source, with clear preferences (file_path for local files and large datasets, content/rows for small datasets under 2,000 rows). It also references alternative upload methods (web app, SDK, CLI) and points to create_scenario_version for updates.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

switch_tenantSwitch TenantB
Read-onlyIdempotent
Inspect

Change which Okareo organization your session operates against.

    Organization selection now happens **during sign-in** (feature 030):
    when you connect the Okareo MCP you choose which organization to
    authorize, and the credential this session uses is already scoped to
    it. This tool therefore no longer changes the active organization — to
    switch, reconnect/re-authenticate the Okareo MCP from your copilot and
    select a different organization when prompted. Use ``list_tenants`` to
    see which organization is currently active.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
tenant_idYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

B3.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description clearly discloses that the tool no longer changes the active organization and explains why (feature 030, credential scoped at sign-in). This goes beyond the annotations (readOnlyHint, destructiveHint, idempotentHint) by providing historical context and the reason for the no-op behavior. It stops short of describing what the tool returns or whether it throws an error, but the core behavioral trait is well covered.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single, focused paragraph that explains the deprecation and provides alternatives. While it includes some background detail (feature number), every sentence serves to prevent misuse. It is appropriately sized for a deprecated tool, though it could be slightly tightened by removing the parenthetical 'feature 030'.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given that the tool is essentially deprecated, the description covers the necessary context: it explains the change, tells the agent how to actually switch, and points to list_tenants. An output schema exists (unseen) which may describe return values, so the lack of return format details in the description is acceptable. It would be more complete if it explicitly stated 'this tool now does nothing' or described the effect of calling it.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 0% description coverage for the single parameter tenant_id, and the description never mentions it. The agent gets no insight into what this parameter is for, how it should be formatted, or whether it is still relevant given the tool's deprecated behavior. The description fails to compensate for the lack of schema documentation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose2/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with 'Change which Okareo organization your session operates against' but immediately contradicts itself by stating 'This tool therefore no longer changes the active organization.' It never clearly states what the tool actually does when invoked now, leaving the agent uncertain whether it is a no-op, a read-only status check, or an error. This is not a tautology but is misleading about the tool's current function.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description explicitly tells the agent to reconnect/re-authenticate the Okareo MCP to switch organizations and to use list_tenants to see the active organization. This provides clear exclusions and alternatives. However, it offers no guidance on when to actually call switch_tenant, making it more of a deprecation notice than a usage guide.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

update_generation_modelUpdate Generation ModelA
Idempotent
Inspect

Change the LLM that a registered generation model points to.

    Updates the generation model to use a different LLM from the registry.
    Use list_available_llms to see available LLMs.

    Args:
        name: Name of the registered generation model to update.
        model_name: The new LLM from the registry.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
nameYes
model_nameYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already indicate readOnlyHint=false, idempotentHint=true, and destructiveHint=false. The description adds useful context by clarifying that the update changes the model to point to a different LLM 'from the registry' and implies that the new model must come from list_available_llms. This is sufficient for a simple update operation, though it does not discuss error cases or validation behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and well-structured: an opening summary sentence, a practical pointer to a related tool, and a clean Args section. Every sentence serves a purpose, and there is no redundant or filler content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple two-parameter update tool that has an output schema and good annotations, the description is complete. It states what the tool does, the source of valid values, and explains both parameters. No further behavioral or return-value detail is needed given the low complexity and available structured metadata.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema provides no descriptions (0% coverage), so the description must compensate. The Args section clearly explains both parameters: 'name' is the registered generation model to update, and 'model_name' is the new LLM from the registry. This adds real meaning beyond the bare property names, even if it does not specify value formats.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description opens with a clear, specific action: 'Change the LLM that a registered generation model points to.' This precisely identifies the verb (change/update), the resource (generation model), and the specific attribute being modified (LLM). It clearly distinguishes this from sibling tools like register_generation_model, get_generation_model, and delete_generation_model.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description provides a clear usage hint: 'Use list_available_llms to see available LLMs.' This guides the agent to a prerequisite/related tool. It does not explicitly state when not to use this tool or list alternative update scenarios, so it falls just short of a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

update_voice_integrationUpdate Voice IntegrationA
Idempotent
Inspect

Update a voice provider integration's metadata.

    Args:
        integration_id: The integration's id.
        metadata: The new metadata object.
    
ParametersJSON Schema
NameRequiredDescriptionDefault
metadataYes
integration_idYes

Output Schema

ParametersJSON Schema
NameRequiredDescription
resultYes

TDQS

A3.5/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare idempotentHint=true, readOnlyHint=false, and destructiveHint=false, but the description adds no extra behavioral context beyond restating the update operation. It does not explain whether the metadata is fully replaced or merged, or any side effects, providing minimal value over the annotations.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is concise and to the point, with a clear opening sentence followed by a minimal Args section. There is no redundancy or wasted wording, making it easy to scan and understand.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

While the core purpose and parameters are covered, the description omits important contextual details like merge-vs-replace behavior, potential constraints on metadata, or prerequisites. Given the output schema exists, return values are not needed, but the description is otherwise minimally adequate for a simple update tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema coverage is 0%, so the description is responsible for explaining parameters. It provides basic definitions for integration_id and metadata, but lacks deeper semantics such as the expected structure of the metadata object or whether it is a partial update, leaving significant room for clarification.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the action ('Update') and the target resource ('a voice provider integration's metadata'), using a specific verb+resource structure. It distinguishes itself from sibling voice integration tools like connect, rotate, and delete by focusing on metadata updates.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance is provided on when to use this tool versus alternatives. The usage is only implied by the tool name and description, offering no exclusions or alternative recommendations.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 51 tool updatesv0.0.43
    • First observedconnect_voice_integration
    • First observedcreate_or_update_check
    • First observedcreate_or_update_driver
    • First observedcreate_or_update_target
    • First observedcreate_scenario_version
    • First observeddelete_check
    • First observeddelete_dashboard
    • First observeddelete_generation_model
    • First observeddelete_scenario
    • First observeddelete_target
    • First observeddelete_voice_integration
    • First observedgenerate_check
    • First observedget_check
    • First observedget_conversation_transcript
    • First observedget_dashboard
    • First observedget_docs
    • First observedget_driver
    • First observedget_generation_model
    • First observedget_reps_baseline
    • First observedget_scenario
    • First observedget_target
    • First observedget_templates
    • First observedget_test_run_results
    • First observedget_voice_integration
    • First observedget_voice_webhook_url
    • First observedingest_conversations
    • First observedlist_available_llms
    • First observedlist_checks
    • First observedlist_dashboards
    • First observedlist_driver_voices
    • First observedlist_drivers
    • First observedlist_generation_models
    • First observedlist_scenarios
    • First observedlist_simulations
    • First observedlist_targets
    • First observedlist_tenants
    • First observedlist_test_runs
    • First observedlist_voice_integrations
    • First observedpreview_delete_scenario
    • First observedquery_analytics
    • First observedreevaluate_test_run
    • First observedregister_generation_model
    • First observedreorder_dashboards
    • First observedrotate_voice_integration_secret
    • First observedrun_simulation
    • First observedrun_test
    • First observedsave_dashboard
    • First observedsave_scenario
    • First observedswitch_tenant
    • First observedupdate_generation_model
    • First observedupdate_voice_integration

TDQS

A3.7/5.0

Scored across 51 tools

Disambiguation4/5

Most tools are clearly distinct by resource and action, with only a few potentially confusing pairs like run_test vs run_simulation or register_generation_model vs create_or_update_target with a 'generation' type. Detailed descriptions and distinct parameters largely resolve ambiguity, so an informed agent can reliably select the right tool.

Naming Consistency4/5

The set predominantly follows a verb_noun pattern (list_, get_, create_or_update_, delete_) with consistent snake_case. Minor deviations exist, such as save_scenario/save_dashboard instead of create_, register_generation_model instead of create_generation_model, and longer phrases like get_voice_webhook_url, which keep it from being perfectly uniform.

Tool Count2/5

At 51 tools, the server is well beyond the 25+ threshold considered too many for a single MCP server. While the broad scope of the Okareo platform explains the abundance, the sheer size will overwhelm agents and make tool selection and discovery unnecessarily difficult.

Completeness3/5

The server covers a wide range of resources with robust CRUD support, but there are notable gaps: there is no delete_driver tool, and no way to delete test runs. The driver lifecycle in particular is incomplete, which is a significant omission for an otherwise comprehensive platform surface.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers