Okareo MCP Server
OfficialOkareo MCP server lets AI coding assistants create/manage evaluation scenarios, models, checks, simulations, voice monitoring, projects, dashboards, and query Okareo docs/analytics directly from the editor.
Scenarios: save, list, get, version, preview/delete input/result datasets.
Generation models: browse LLMs, register/list/get/update/delete models under test.
Tests & checks: run quality tests, view/poll results and transcripts, re-evaluate runs, create/generate/get/delete checks, calibrate draft checks.
Simulations: create/manage targets (generation, custom endpoint, voice/Twilio/SIP) and driver personas, discover voices, run/list multi-turn simulations, voice augmentations.
Voice monitoring: ingest conversations, connect/manage voice provider integrations (Retell, Twilio, VAPI, ElevenLabs), rotate secrets, get webhook URLs.
Projects: list, select, create, update, archive/unarchive, clone projects (no delete).
Analytics & dashboards: query product analytics, list/get/save/reorder/delete dashboards.
Tenant management (remote OAuth only): list tenants; switch via re-auth.
Docs & templates: query Okareo docs, fetch offline prompt templates, serve REPS agent-evaluation baseline material.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@Okareo MCP Serverrun evaluation on my latest model"
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
Okareo MCP Server
The Okareo MCP server exposes Okareo's evaluation capabilities as MCP tools, allowing AI coding assistants to create and manage scenarios, register models, run evaluations, and execute multi-turn simulations directly from your editor.
For detailed documentation, see the Okareo MCP docs.
Prerequisites
An Okareo account at app.okareo.com
A copilot that supports MCP servers (Claude Code, Cursor, or VS Code)
Python 3.10–3.12 (only for the local install modes; not needed for remote)
Related MCP server: Patronus MCP Server
Remote MCP (hosted) — recommended
The fastest way to get started. No local install, no Python, no API key in .mcp.json. Browser sign-in handles auth on first connect.
Recommended (OAuth — Claude Code, Claude Desktop, Cursor, VS Code 1.101+)
Add to your copilot's MCP config (typically .mcp.json):
{
"mcpServers": {
"okareo": {
"type": "http",
"url": "https://tools.okareo.com/mcp"
}
}
}Reload the copilot. It will open a browser to Okareo sign-in once; thereafter the copilot stores the token itself. Your .mcp.json contains no secrets.
Fallback (Bearer header — older clients or headless / CI)
For clients that haven't shipped the MCP OAuth flow yet, paste your API key as a bearer header. Prefer the env-var form over an inline literal:
{
"mcpServers": {
"okareo": {
"type": "http",
"url": "https://tools.okareo.com/mcp",
"headers": {
"Authorization": "Bearer ${env:OKAREO_API_KEY}"
}
}
}
}Working across multiple Okareo organizations
If you have access to more than one Okareo account, simply Clear Authentication and then re-authenticate with the account you want to work in. You can always ask which account your are in and have acccess to.
Local install (alternative)
Run the MCP server on your own machine. Useful for offline / airgapped environments and for development.
Prefer the Remote MCP section above unless you have a specific reason to install locally (airgapped, custom build, development on this repo). The remote endpoint requires no install and stays current automatically.
For multi-org users, the remote endpoint also exposes tenant management tools (list_tenants, switch_tenant).
Step 1: Set Your API Key
export OKAREO_API_KEY="your-api-key"Add this to your ~/.zshrc or ~/.bash_profile for persistence.
Step 2: Configure Your Copilot
Claude Code
Add to .mcp.json:
{
"mcpServers": {
"okareo": {
"command": "uvx",
"args": ["okareo-mcp"],
"env": {
"OKAREO_API_KEY": "${OKAREO_API_KEY}"
}
}
}
}No pre-install needed — uvx handles it automatically.
Cursor
Add to .cursor/mcp.json:
{
"mcpServers": {
"okareo": {
"command": "uvx",
"args": ["okareo-mcp"]
}
}
}Cursor inherits OKAREO_API_KEY from your shell environment.
Alternative: pip install
If you don't have uv installed:
pip install okareo-mcpThen use "command": "okareo-mcp" instead of "command": "uvx" with "args": ["okareo-mcp"].
Configuration Reference
Variable | Default | Description |
| (required) | Your Okareo API key |
|
| Override for on-prem Okareo backend |
|
| Transport: |
|
| Port for SSE transport |
Available Tools
Scenarios
Tool | Description |
| Save a named scenario from rows of input/result data (idempotent) |
| List all scenarios in the project with names, IDs, and row counts |
| Retrieve a scenario's metadata and all data rows by name or ID |
| Create a new version of an existing scenario with updated data |
| Preview what will be deleted before removing a scenario |
| Permanently delete a scenario and all related test data |
Generation Models
Tool | Description |
| Browse available LLMs from the Okareo registry |
| Register a generation model for testing by selecting an LLM from the registry |
| List all registered generation models in the project |
| Read detailed information about a registered generation model |
| Change the LLM a registered generation model points to |
| Remove a registered generation model and all its related test data |
Tests & Checks
Tool | Description |
| List available quality checks (built-in and custom) for evaluating model outputs |
| Run a quality test that evaluates a model against a scenario using specified checks |
| List past test runs with optional filters (model, scenario, simulation-only) |
| Load detailed per-row results of a test run or simulation by ID or name |
| Retrieve the full conversation transcript for a single data point |
| Re-score a completed test run against a (possibly different) set of checks |
| Create or update a quality check by name — model-based, code-based, or audio (upsert) |
| Generate a check from a natural-language description, then save it |
| Retrieve a check's full configuration, including its prompt template or code |
| Permanently delete a check by name |
| Dry-run a draft check against a finished test run — per-row verdicts and the arguments the check received, saving nothing |
Simulations (Multi-Turn)
Tool | Description |
| Create or update a Target — generation model, custom endpoint, or voice (OpenAI, Deepgram, Twilio) |
| Retrieve a Target's configuration by name (all types) |
| List all simulation targets (voice and custom_endpoint) in the project |
| Remove a simulation target and all its related test data |
| Define a simulated user persona that will interact with your target |
| Retrieve a Driver's full configuration including the persona prompt |
| List all Driver personas in the project |
| Discover the voices, voice profiles, and languages available for voice drivers |
| Run a multi-turn conversation evaluation (or rerun a previous one with overrides) |
| List past simulation runs with optional filters (target, scenario, limit) |
Voice Monitoring
Tool | Description |
| Submit completed voice conversations to Okareo for monitoring |
| Connect a voice provider so its traffic flows into Okareo monitoring |
| List the voice provider integrations in your project |
| Retrieve a voice provider integration by ID, including its status |
| Update a voice provider integration's metadata |
| Rotate a voice provider integration's secrets |
| Delete a voice provider integration by ID |
| Get the inbound webhook endpoint for a voice provider |
Projects
Tool | Description |
| List every project you can work in — id, name, tags, archive state — with the active one marked |
| Choose the project to work in for this conversation |
| Create a new project |
| Rename a project, or replace its tags |
| Hide a project from the project picker. Reversible; nothing is deleted |
| Restore an archived project to the picker |
| Copy an existing project's scenarios into a new project (dry-run first) |
Okareo has no project delete — in the app, the API, or here. Archiving is the removal, and it only hides the project from the picker: its scenarios, runs, and dashboards stay intact and every tool still works against it.
Analytics & Dashboards
Tool | Description |
| Query Okareo's product analytics to understand evaluation trends |
| List the analytics dashboards in your project |
| Retrieve a dashboard's full configuration by name |
| Create or update an analytics dashboard by name (upsert) |
| Set the display order of dashboards |
| Delete a dashboard by name |
Tenant Management (remote MCP only)
Tool | Description |
| List every Okareo organization you have access to in this MCP session |
| Change which Okareo organization subsequent tool calls operate against |
Documentation & Templates
Tool | Description |
| Query the Okareo documentation system for conceptual or user-legible explanations |
| Retrieve prompt templates for common Okareo patterns (works offline) |
| Serve REPS agent-evaluation baseline material (scenario banks, drivers, checks, eval configs) from the latest tagged okareo-tools release |
Troubleshooting
Symptom | Cause | Fix |
| Not installed or not in PATH | Run |
Server exits with API key error |
| Export it: |
| Okareo SDK requires Python <3.13 | Use Python 3.10–3.12 |
Copilot can't connect (Docker) | Wrong URL | Ensure URL ends with |
Cursor doesn't pick up API key | Cursor launched from Dock, not terminal | Launch Cursor from terminal: |
Contributing
This repository is a curated public mirror; the canonical source is maintained by Okareo. We welcome issues and consider community pull requests — see CONTRIBUTING.md for how proposed changes are reviewed and ported.
License & Trademarks
The Okareo MCP server source code is licensed under the Apache License 2.0 (see also NOTICE).
"Okareo", the Okareo logo, and Okareo product names are trademarks of Okareo, Inc. and are not covered by the Apache 2.0 license. See TRADEMARK.md for permitted use.
Available Tools
51 toolsconnect_voice_integrationConnect Voice IntegrationAInspect
Connect a voice provider so its traffic flows into Okareo monitoring.
Creates a provider integration. The returned integration carries an id
and a public_id — pass the provider + public_id to get_voice_webhook_url
to obtain the inbound webhook endpoint to paste into the provider's
console.
Args:
provider: Voice platform — one of: retell, twilio, vapi, elevenlabs.
webhook_auth_type: Webhook authentication type expected by Okareo
for this provider (provider-specific — see Okareo docs).
secrets: Provider-specific secret values (opaque pass-through; the
response never echoes raw secrets, only a summary).
metadata: Optional free-form metadata object.
| Name | Required | Description | Default |
|---|---|---|---|
| secrets | Yes | ||
| metadata | No | ||
| provider | Yes | ||
| webhook_auth_type | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations, the description adds meaningful behavioral details: the response carries an id and public_id, secrets are never echoed and only summarized, and webhook_auth_type is provider-specific. This helps the agent understand side effects and security aspects. No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is about 150 words and well-structured, with a clear purpose statement followed by parameter explanations. Every sentence adds value; no fluff or repetition. It's slightly longer than necessary but appropriate given the parameter complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description covers the main success behavior (creation, returned id/public_id) and directs to the next step (get_voice_webhook_url). With an output schema present, it doesn't need to enumerate all return fields. It also covers all parameters and provider options, making it sufficiently complete for this mutating tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage, so the Args section in the description carries the full burden. It adds substantial meaning: lists allowed provider values (retell, twilio, vapi, elevenlabs), explains webhook_auth_type is provider-specific, describes secrets as opaque pass-through, and notes metadata is optional. This fully compensates for the missing schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action: "Connect a voice provider so its traffic flows into Okareo monitoring" and "Creates a provider integration." This is a specific verb+resource, and it distinguishes from sibling tools like update_voice_integration or rotate_voice_integration_secret by focusing on initial setup.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use it (connecting a voice provider) and even mentions the follow-up step of using get_voice_webhook_url with the returned public_id. It doesn't explicitly state exclusions or alternatives, but the focus on creation makes the usage fairly obvious.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
create_or_update_checkCreate or Update CheckAIdempotentInspect
Create or update a quality check by name (upsert). Supports model-based, code-based, and audio checks.
Saving to an existing name creates a new version of that check (see
get_check's "available_versions"). Before writing a prompt_template or
code_contents from scratch, fetch a worked example with get_templates:
"boolean_check_prompt", "score_check_prompt", "analysis_check_prompt",
or "check_code".
Args:
name: Unique name for the check.
description: What the check evaluates.
check_type: "model" (an LLM judge driven by prompt_template) or
"code" (a deterministic Python class in code_contents).
output_type: "pass_fail" (boolean verdict), "score" (numeric, e.g.
a 1-5 rubric), or "analysis" (free-form qualitative feedback;
only valid with check_type="model"). For check_type="code" the
server infers pass_fail vs score from the value evaluate()
returns (bool vs int/float) — output_type is used only to
validate the request, not sent to the server. Note: list_checks
and get_check report this as output_data_type in the server
vocabulary, where "bool" means pass_fail and "int" means score.
prompt_template: Required when check_type="model". The judge
prompt. Inject the runtime data the judge needs with these
placeholders:
- {model_output}: the model output being evaluated. In a
multi-turn conversation this is ONLY the final assistant
message, not the full conversation.
- {scenario_input}: the scenario input / source text.
- {scenario_result}: the reference/expected output.
- {model_input}: what was sent to the model (prompt or
messages).
- {message_history}: the full multi-turn conversation — the
model_input messages plus the assistant's model_output. Use
this when the check must judge the whole conversation.
- {tool_calls}: the tool/function calls the model just made.
- {tools}: the tool definitions/schema available to the model.
- {model_output_metadata}: metadata attached to the most
recent model output.
- {simulation_message_history}: full conversation history
reconstructed from trace metadata. Only populated for traced
(ingested) conversations; for simulations and evaluations
use {message_history}.
The legacy {generation} placeholder is deprecated — use
{model_output} instead.
code_contents: Required when check_type="code" (output_type
"pass_fail" or "score" only). Python source defining
`class Check(CodeBasedCheck)` with a
`@staticmethod def evaluate(...) -> CheckResponse` method.
Start from `from okareo.checks import CodeBasedCheck,
CheckResponse`. evaluate() may declare any subset of these
parameters: model_output, scenario_input, scenario_result,
metadata, model_input. Return CheckResponse(score=...,
explanation=...) where score is a bool for pass_fail or an
int/float for score. See get_templates("check_code") for
complete examples.
is_audio: Set to true for audio/voice evaluation. Only valid with
check_type="model".
tags: Optional list of string tags to organize the check. Tags are
stored with the check and returned by get_check.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | ||
| tags | No | ||
| is_audio | No | ||
| check_type | Yes | ||
| description | Yes | ||
| output_type | Yes | ||
| code_contents | No | ||
| prompt_template | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
While annotations already indicate idempotency (idempotentHint=true), the description adds crucial behavioral nuances: saving to an existing name creates a new version, output_type for code checks is only validated client-side and not sent to the server, and the server-side vocabulary difference (output_data_type) is explained. It also notes the deprecated placeholder, providing transparency beyond what annotations offer.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but exceptionally well-structured. It leads with the main purpose, then organizes parameter details in an 'Args' block with clear formatting. The extensive placeholder list and code examples are essential for a tool with this complexity, and every section carries meaningful information without repetition.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (8 parameters, model vs code check variants, audio support, placeholder injection), the description is remarkably complete. It covers prerequisites (get_templates), versioning behavior, parameter interdependencies, code class structure, and server-side nuances. Since an output schema is present, the absence of return-value explanation is acceptable.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, but the description fully compensates by explaining each parameter in depth. It details valid values, dependencies (e.g., prompt_template required for model checks), placeholder semantics for prompt_template, and the expected Python structure for code_contents. This far exceeds the schema's bare type definitions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Create or update a quality check by name (upsert).' It clearly distinguishes from siblings like get_check, delete_check, and generate_check by focusing on the creation/update operation. The scope (model, code, audio) is explicit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use this tool and when to consult alternatives. It explicitly directs users to get_templates for worked examples before writing prompt_template or code_contents from scratch, and references get_check's 'available_versions' for versioning behavior. It does not explicitly exclude alternative tools like generate_check, but the contextual guidance is strong.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
create_or_update_driverCreate or Update DriverAIdempotentInspect
Define a simulated user persona that will interact with your target.
Creates or updates a Driver by name (upsert). Author ONLY the core
persona sections in prompt_template, in this order:
- `## Persona` — who the simulated user is (static character).
- `## Scenario Details` — contains the scenario reference
(`{scenario_input}` or a specific path like
`{scenario_input.objectives}`), placed immediately before
Objectives. This is how each scenario row's data reaches the
conversation.
- `## Objectives` — WHAT the driver is trying to accomplish, written
from the driver's goal (not from scenario variables).
- `## Soft Tactics` — HOW the driver probes, escalates, and stops.
Do NOT author Hard Rules, a Turn-End Checklist, or Conversation
Behavior sections: the MCP automatically appends the platform's
canonical versions of those blocks (including the language rule
matching `language`) — the same blocks the Okareo UI appends to
generated drivers. Any caller-authored variant of these sections is
replaced by the canonical text, and repeated updates never duplicate
the blocks.
For voice agents, configure how the simulated user speaks with `voice`,
`voice_profile`, `voice_instructions`, and `language`. Call
list_driver_voices first to discover valid voice and profile values.
Args:
name: Unique name for this driver.
prompt_template: The core persona prompt (Persona, Scenario
Details, Objectives, Soft Tactics — see above). Hard Rules and
Conversation Behavior are appended automatically.
model_id: Foundation model to power the driver (defaults to project default).
temperature: Response randomness, default 0.6.
voice_instructions: Free-text speaking instructions for voice simulations
(tone, pace, accent). Not validated against the voice catalog.
voice_profile: Voice profile name for voice simulations. Validated
against the catalog from list_driver_voices.
voice: Voice identifier for voice simulations. Validated against the
catalog from list_driver_voices.
language: Language the driver responds in, as the bare ISO code
the voice catalog serves (e.g. "en", "es", "ja"); regional
variants like "fr-CA" are accepted when their base code
matches the voice's language. When a `voice` is set and
language is omitted, it is derived from that voice's catalog
language (disclosed as `language_derived_from_voice` in the
response); a value conflicting with the voice's language is
rejected. Also drives the appended Hard Rules language rule.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | ||
| voice | No | ||
| language | No | ||
| model_id | No | ||
| temperature | No | ||
| voice_profile | No | ||
| prompt_template | Yes | ||
| voice_instructions | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations (idempotentHint=true), the description discloses that caller-authored canonical sections are replaced and repeated updates never duplicate blocks. It also reveals language conflict rejection and language derivation behavior with voice, which are non-obvious behavioral traits not inferable from annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is organized with an opening summary, explicit authoring rules, voice configuration guidance, and a per-parameter breakdown. Every sentence adds value, and the structure makes scanning easy despite the length.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description is fully complete for a complex tool: it covers all parameters, prerequisites, validation behavior, and automatic append behavior. It even mentions response disclosure (language_derived_from_voice). With an output schema available, no return description is needed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The 'Args:' section adds rich semantics for all 8 parameters: uniqueness of name, allowed prompt_template sections, model_id default, temperature default, voice validation against the catalog, and language rules including regional variants and derivation. This goes far beyond the bare schema types and defaults.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a clear summary: 'Define a simulated user persona that will interact with your target. Creates or updates a Driver by name (upsert).' This is a specific verb+resource statement that distinguishes the tool from siblings like get_driver, list_drivers, and list_driver_voices.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives explicit usage guidance, including a 'Do NOT author Hard Rules...' warning and instructs callers to 'Call list_driver_voices first to discover valid voice and profile values.' This tells when to use a sibling tool first and what not to include, which is explicit when/when-not guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
create_or_update_targetCreate or Update TargetAIdempotentInspect
Create or update a Target — the AI system you want to evaluate in a simulation.
Calling create_or_update_target with the same name as an existing Target will
**fully replace** its configuration — caller must re-specify all desired fields.
Supported types: 'generation' (foundation model), 'custom_endpoint' (your own
REST API), and 'voice' (voice-based targets reached by phone via Twilio or
over SIP).
**Cloning workflow**: this tool accepts the same key structure that `get_target`
returns, so you can read an existing Target, change `name`, swap in real values
for any field whose value is `"***REDACTED***"`, and pass the result here as
kwargs. Calls that still contain the redaction sentinel are rejected with an
error naming each offending path; the sentinel is never forwarded to the backend.
Args:
name: Unique name for this target.
type: Target type — 'generation', 'custom_endpoint', or 'voice'.
model_id: (generation targets) Foundation model ID, e.g. 'gpt-4o-mini'.
temperature: (generation targets) Response randomness, default 0.
system_prompt_template: (generation targets) System instructions; mustache
syntax supported, e.g. '{scenario_input}'.
user_prompt_template: (generation targets) User prompt template.
dialog_template: (generation targets) Dialog formatting template.
tools: (generation targets) Tool definitions for function calling.
next_message_params: (custom_endpoint) Nested HTTP config for each
conversation turn. Required keys: 'url', 'method'. Optional:
'headers', 'body', 'status_code', 'response_message_path',
'response_session_id_path', 'response_tool_calls_path'.
All response path values MUST use dot-path notation starting with
'response.' — e.g., 'response.message', 'response.choices[0].message.content',
'response.choices[0].tool_calls'. Never use bare property names.
For SSE/streaming endpoints, include a 'streaming' object with:
- 'stop': array of stop conditions (OR semantics — any match ends
the stream). Each has 'value' (required) and optional 'path'
(dot-path into JSON chunk). Without 'path', matches raw SSE data.
- 'select': array of select conditions (AND semantics — all must
match for a chunk's content to be extracted). Each requires
'path' and 'value'.
When streaming, set response_message_path to the chunk field
(e.g., 'response.choices[0].delta.content').
start_session_params: (custom_endpoint, optional) Nested HTTP config to
initialise a session. Required key: 'url'. Optional: 'method',
'headers', 'body', 'status_code', 'response_session_id_path'
(dot-path starting with 'response.', e.g. 'response.id'),
'response_message_path'. Supports 'streaming' object (same
structure as next_message_params.streaming).
end_session_params: (custom_endpoint, optional) Nested HTTP config to
close a session after the last turn.
auth_params: (custom_endpoint, optional) Token-based authorization config.
Required keys when provided: 'url', 'method', 'response_access_token_path'
(dot-path starting with 'response.', e.g. 'response.access_token').
Optional: 'headers', 'body', 'status_code'.
sensitive_fields: (custom_endpoint, optional) List of dot-path strings for
secret fields within auth_params (e.g., 'auth_params.body.client_id').
The MCP auto-generates entries for top-level auth_params keys; use this
for deeper paths. To remove auth from an existing target, call
create_or_update_target again without auth_params.
max_parallel_requests: (custom_endpoint, twilio) Concurrency limit. This is
the same setting the Okareo web UI labels "max concurrency".
edge_type: (voice targets) How Okareo reaches the voice agent —
'twilio' (dial a phone number) or 'sip' (call a SIP URI).
to_phone_number: (voice twilio) Destination phone number (required).
account_sid: (voice twilio, custom only) Twilio account SID. If provided,
auth_token and from_phone_number are also required (all-or-nothing).
Omit for generic Twilio targets using Okareo's managed integration.
auth_token: (voice twilio, custom only) Twilio auth token. Required with
account_sid and from_phone_number.
from_phone_number: (voice twilio, custom only) Caller phone number. Required
with account_sid and auth_token.
sip_uri: (voice sip) Destination SIP URI (required), e.g.
"sip:agent@your-domain.example.com". Use this to test any voice
agent reachable over SIP — for example one fronted by Daily,
Vapi, LiveKit, or a SIP trunk.
sip_username: (voice sip, optional) SIP authentication username.
sip_password: (voice sip, optional) SIP authentication password
(stored as a sensitive field).
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | ||
| type | Yes | ||
| tools | No | ||
| sip_uri | No | ||
| model_id | No | ||
| edge_type | No | ||
| auth_token | No | ||
| account_sid | No | ||
| auth_params | No | ||
| temperature | No | ||
| sip_password | No | ||
| sip_username | No | ||
| dialog_template | No | ||
| to_phone_number | No | ||
| sensitive_fields | No | ||
| from_phone_number | No | ||
| end_session_params | No | ||
| next_message_params | No | ||
| start_session_params | No | ||
| user_prompt_template | No | ||
| max_parallel_requests | No | ||
| system_prompt_template | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations (idempotentHint=true, destructiveHint=false), the description discloses critical behaviors: 'fully replace its configuration — caller must re-specify all desired fields', redaction sentinel rejection, auto-generation of sensitive fields, and the all-or-nothing rule for Twilio auth. These details significantly exceed what the annotations alone convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but justified by the tool's complexity (22 parameters, three target types). It uses clear sectioning (type-specific parameter groups) and front-loads the core purpose and replacement warning. A few sentences, like the streaming stop/select details, are dense but necessary. Very little is wasted.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a tool with 22 parameters, 0% schema descriptions, and a rich output schema, the description is fully complete. It covers all parameter categories, type-specific configurations, redaction handling, replacement semantics, and auth removal. The presence of an output schema means return-value documentation is unnecessary, and the description fills all other gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema coverage, the description carries the full burden and excels: every one of the 22 parameters is documented with type-specific context, requiredness, defaults, examples, and nested structures (e.g., next_message_params with dot-path rules and streaming stop/select semantics). This adds enormous meaning beyond the bare schema property names.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb+resource: 'Create or update a Target — the AI system you want to evaluate in a simulation.' It clearly distinguishes from sibling tools like get_target, list_targets, and delete_target by covering both creation and update in one operation. The title and description align perfectly.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear usage context, including the cloning workflow ('this tool accepts the same key structure that `get_target` returns') and the replacement behavior when reusing a name. It also explains type-specific requirements (generation, custom_endpoint, voice) and how to remove auth. However, it does not explicitly state when to use an alternative tool, instead relying on the context of the cloning workflow.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
create_scenario_versionCreate Scenario VersionAInspect
Create a new version of an existing scenario with updated data.
Automatically determines the next version number (e.g., 'my-test-v2',
'my-test-v3'). The original scenario is treated as version 1.
Args:
base_name: Name of the original scenario to create a version of.
rows: List of data rows for the new version, each with 'input' and 'result'.
| Name | Required | Description | Default |
|---|---|---|---|
| rows | Yes | ||
| base_name | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations only indicate it is a non-read-only, non-idempotent, non-destructive operation. The description adds useful behavioral details: it automatically determines the next version number (e.g., 'my-test-v2'), treats the original as version 1, and specifies the row structure. However, it does not disclose error behavior (e.g., if base_name does not exist) or side effects beyond creation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is efficiently structured: a clear one-line summary, followed by a concise explanation of version numbering and an Args list. The Args section is useful and not overly verbose. It earns its place, though the docstring format is slightly technical.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool is simple with two required parameters and an output schema, so the description covers the essentials: purpose, versioning behavior, and parameter semantics. However, it lacks usage guidelines relative to alternatives and does not address prerequisites or error cases, leaving some contextual gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema description coverage, the description compensates by explaining both parameters: base_name is the original scenario name, and rows are data rows with 'input' and 'result' fields. This adds meaning beyond the schema's bare titles and types, though it could further clarify required field types or whether rows must contain exactly those keys.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: 'Create a new version of an existing scenario with updated data.' It uses a specific verb+resource combination and distinguishes it from siblings like save_scenario by emphasizing versioning and automatic version-number determination.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage context (creating new versions with auto-incremented numbering) but does not explicitly state when to use it over alternatives like save_scenario, nor does it provide exclusions or conditions. The behavior of generating the next version number is a clear signal, but no direct usage guidance is given.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
delete_checkDelete CheckADestructiveInspect
Permanently delete a check by name.
Args:
name: Name of the check to delete.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description adds the word 'permanently', which is a behavioral trait not explicitly stated in the annotations. However, it does not disclose any other side effects, failure modes, prerequisites, or what happens to associated data. The destructiveHint annotation already signals destructiveness, so the description's contribution is minimal but present.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is extremely concise, consisting of a single clear sentence and an Args block. Every element is necessary and there is no wasted text. It is well-structured and front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple single-parameter destructive operation with annotations indicating destructiveness and an output schema, the description is adequate. The 'permanently' keyword signals irreversibility, and the parameter is explained. It doesn't detail return values, but the output schema likely covers that. Given the tool's simplicity, the description is nearly complete, though it could mention error conditions (e.g., check not found).
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has no description for the 'name' parameter (0% coverage), so the description provides the meaning by stating 'Name of the check to delete'. This clarifies the parameter's role, but it does not specify whether the name is a unique identifier, case-sensitive, or how it relates to names in other tools. The description compensates for the schema gap but only at a basic level.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb 'delete' with a specific resource 'check' and adds 'permanently' to clarify the action's scope. It clearly distinguishes this from sibling tools like get_check, list_checks, and create_or_update_check.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no explicit guidance on when to use this tool versus alternatives. It does not mention checking existence first, using preview_delete_scenario for checks, or any when-not-to-use conditions. The usage is only implied by the verb 'delete'.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
delete_dashboardDelete DashboardBDestructiveInspect
Delete a dashboard by name.
Args:
name: Name of the dashboard to delete.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotation already declares destructiveHint=true. The description adds no extra behavioral context such as permanence, irreversibility, or required permissions. It simply restates the basic operation without enriching the safety profile.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is very short and front-loaded with the main action. It avoids unnecessary verbiage, though it could have been slightly more informative without becoming verbose.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the single parameter and the destructive annotation, the description is minimally sufficient but lacks context about error handling (e.g., what happens if the dashboard doesn't exist) or any side effects. For a destructive operation, slightly more detail would improve completeness.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has zero description coverage, and the description only repeats the parameter name ('name: Name of the dashboard to delete'). It adds no additional meaning about format, uniqueness, or edge cases beyond what the schema field name already implies.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Delete') and the specific resource ('a dashboard') with a required parameter ('by name'). This distinguishes it from sibling delete tools that target other resources (e.g., delete_scenario, delete_generation_model).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No guidance is provided about when to use this tool versus alternatives. It does not mention any prerequisites, exclusions, or contextual conditions for deletion.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
delete_generation_modelDelete Generation ModelADestructiveInspect
Remove a registered generation model and all its related test data.
Permanently deletes the generation model and cascades to associated
test runs and test data points. This cannot be undone.
Args:
name: Name of the registered generation model to delete.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description goes beyond the annotation's destructiveHint by detailing specific behavioral traits: it 'cascades to associated test runs and test data points' and is irreversible ('This cannot be undone'). This adds meaningful context about the destructive scope that the annotation alone does not convey.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and front-loaded with the action. It includes two brief sentences and an Args list, all of which add value. It could be slightly tighter by merging the first two sentences, but there is no waste and the structure is clear.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple delete tool with one required parameter, the description is nearly complete. It covers the action, cascading effects, irreversibility, and the parameter meaning. The presence of an output schema and annotations reduces the need to describe return values or safety flags, making this sufficiently comprehensive.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema only provides the parameter name 'Name' with no description. The description's Args section supplies semantic meaning: 'Name of the registered generation model to delete.' This clarifies exactly what value is expected, fully compensating for the 0% schema description coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly identifies the tool's purpose: 'Remove a registered generation model and all its related test data.' It uses a specific verb ('remove') and resource ('generation model'), and explicitly mentions the cascading scope. This distinguishes it from sibling tools like update_generation_model or get_generation_model.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies the use case (deleting a model) and includes a strong caution ('This cannot be undone'), but it does not explicitly state when to use this versus alternatives, nor does it mention any prerequisites like listing models first. There are no exclusions or when-not-to-use guidance, so it is adequate but not highly instructive.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
delete_scenarioDelete ScenarioADestructiveInspect
Permanently delete a scenario and all related test data.
Both scenario_id and name are required. Use preview_delete_scenario first
to see what will be deleted before confirming.
Args:
scenario_id: The ID of the scenario to delete (from preview_delete_scenario).
name: The name of the scenario to delete.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | ||
| scenario_id | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark destructiveHint=true, and the description enhances this by stating 'permanently delete' and 'all related test data', clarifying the scope and irreversibility. It also adds the safety practice of previewing, going beyond the structured annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is short, front-loaded with the core action, and presents usage guidance and parameter details efficiently without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a destructive tool with an output schema, the description sufficiently covers prerequisites, the destructive scope, and the recommended preview step, making it complete for its complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema provides basic names/types, and the description adds context for scenario_id (from preview_delete_scenario) and reiterates that both are required. With 0% schema coverage, it partially compensates but could offer more detail on how to obtain or validate the name.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses the specific verb 'delete' with the resource 'scenario' and adds scope ('all related test data'), clearly distinguishing it from sibling tools like preview_delete_scenario and other delete_* tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly instructs to use preview_delete_scenario first before confirming deletion, providing direct guidance on proper tool sequencing and when to invoke this tool.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
delete_targetDelete TargetADestructiveInspect
Remove a simulation target and all its related test data.
Permanently deletes the target and cascades to associated
test runs and test data points. This cannot be undone.
Args:
name: Name of the target to delete.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description adds that deletion is permanent, cascades to associated test runs and data points, and cannot be undone. This exceeds the destructiveHint annotation by detailing the scope and irreversibility, providing valuable behavioral context.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three focused sentences plus a minimal Args block. It front-loads the main action, then adds cascade and irreversibility details, with no redundant text.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple one-parameter delete with an output schema and destructive annotation, the description covers the essential information: what is deleted, cascading behavior, and permanence. It does not mention error behavior for non-existent targets, but the annotation and low complexity make the description adequately complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema description coverage, the description's Args section ('name: Name of the target to delete') provides the only explanation of the parameter, clarifying it is the target's name. However, it lacks additional constraints like uniqueness or how to obtain the name, but for a single-parameter delete tool it is sufficient.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states 'Remove a simulation target and all its related test data' with a clear verb and resource, and further clarifies permanent deletion and cascading to test runs/data points. This distinguishes it from other delete_ siblings by specifying the resource type (simulation target) and side effects.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for what the tool does (cascading deletion of a target) but does not explicitly state when to use it versus alternatives like delete_scenario or preview_delete_scenario. No exclusions or alternative recommendations are given, so usage is implied rather than explicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
delete_voice_integrationDelete Voice IntegrationBDestructiveInspect
Delete a voice provider integration by id.
Args:
integration_id: The integration's id.
| Name | Required | Description | Default |
|---|---|---|---|
| integration_id | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description adds no behavioral detail beyond what the annotations already disclose (destructiveHint=true, readOnlyHint=false). It does not mention permanence, cascading effects, or potential failures, so it contributes no extra transparency.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is very concise and front-loaded with the core action. The Args block duplicates schema information unnecessarily, but the overall structure is efficient and easy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the simple one-parameter operation, the description is adequate. Annotations and output schema cover the destructiveness and return value, but the description lacks context on side effects or when deletion is appropriate. It fits the tool's low complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With schema description coverage at 0%, the description must compensate. It provides a minimal definition ('The integration's id'), which clarifies the parameter's role but adds little beyond the schema's title. This is minimally sufficient for a single simple parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Delete'), the resource ('voice provider integration'), and the scope ('by id'). This distinguishes it from sibling tools like update_voice_integration or list_voice_integrations, which perform different operations on the same resource.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no explicit guidance on when to use this tool versus alternatives (e.g., update_voice_integration) or any prerequisites/exclusions. It only implies usage for deletion, leaving the decision solely to the agent's intent.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
generate_checkGenerate CheckAInspect
Generate a check from a natural language description. Uses AI to create the prompt template (model checks) or Python code (code checks), then saves the check.
Use this when you only have a description of what to evaluate. When
you already know the exact prompt template or Python code the check
should use, call create_or_update_check directly instead. The
generated prompt/code is returned in the response — review it and
refine with create_or_update_check if needed.
Args:
name: Name for the generated check.
description: Natural language description of what to evaluate
(e.g., "check if the response is toxic"). The more specific
the description, the better the generated check.
output_type: "pass_fail" (boolean verdict), "score" (numeric), or
"analysis" (free-form qualitative feedback; model checks only).
check_type: "model" (LLM judge) or "code" (deterministic Python).
requires_scenario_input: Set true when the evaluation must compare
the output against the scenario input. The generated check
will reference {scenario_input} and only works on runs whose
scenarios provide it.
requires_scenario_result: Set true when the evaluation must
compare the output against the expected result. The generated
check will reference {scenario_result} and only works on runs
whose scenarios provide it.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | ||
| check_type | No | model | |
| description | Yes | ||
| output_type | No | pass_fail | |
| requires_scenario_input | No | ||
| requires_scenario_result | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations show readOnlyHint=false, indicating a mutation. The description adds that AI generates the prompt/code, that the generated output is returned for review, and that scenario flags affect which runs the check works on. This goes beyond the bare annotations meaningfully, though it doesn't mention permissions or failure modes.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is somewhat lengthy but well-structured: a clear purpose statement, then usage guidance, then a structured argument list. Every section adds necessary value, especially given the low schema coverage. It could be slightly tightened, but the length is justified by the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description thoroughly covers the tool's purpose, when to use it, all parameters, the generated output returned for review, and dependencies on scenario flags. The presence of an output schema further reduces the need to describe return structure. It also points to create_or_update_check for refinement, providing a complete workflow.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description fully compensates. It explains all six parameters, including the allowed values for output_type ('pass_fail', 'score', 'analysis') and check_type ('model', 'code'), and clarifies the behavior of requires_scenario_input/result with example usage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool generates a check from a natural language description using AI, produces a prompt template or Python code, and saves the check. It explicitly distinguishes from sibling create_or_update_check by contrasting when to use each.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides explicit usage guidance: use this tool when only a description exists, and use create_or_update_check directly when the exact prompt/code is known. It also suggests refining the generated output with create_or_update_check, giving clear when-to and when-not-to context.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_checkGet CheckARead-onlyIdempotentInspect
Retrieve the full configuration of a check by name, including its prompt template or code contents.
Args:
name: Name of the check to retrieve.
version: Optional check version number to pin. Omit (or leave null)
for the most recent version. The response always lists every
available version under "available_versions".
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | ||
| version | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
With readOnlyHint and idempotentHint already covering safety, the description adds valuable behavioral context: the response always includes 'available_versions' and that omitting version returns the most recent. It also discloses that the configuration includes prompt template or code contents, going beyond the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with the main purpose in one sentence, followed by a compact Args block. Every sentence adds necessary context without redundant filler or repetition of schema information.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a get-by-name endpoint with two parameters and an output schema, the description covers the optional version behavior and guarantees that all versions are returned in the response. It does not need to describe return fields because an output schema exists, and annotations handle safety semantics.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Despite 0% schema description coverage, the Args section fully documents both parameters: name identifies the check, and version is optional with pinning behavior and a clear directive to omit for the latest version. This adds meaning well beyond the bare schema types and default.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'Retrieve the full configuration of a check by name,' which clearly states the verb (retrieve), resource (check configuration), and scope (by name). It further distinguishes from list_checks by specifying 'including its prompt template or code contents,' making the specific use case unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage when you need a full check configuration (as opposed to a list), but it does not explicitly name alternatives or state when not to use it. The version guidance is useful but does not address tool selection among siblings like list_checks or get_driver.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_conversation_transcriptGet Conversation TranscriptARead-onlyIdempotentInspect
Retrieve the full conversation transcript for a single data point.
Use this after get_test_run_results to drill into a specific
conversation. Provide either scenario_index (1-based, from the
scores summary) or test_id (UUID) to identify the conversation.
Returns the complete message transcript (model_input), final
output (model_result), per-turn check scores (metric_value),
and the scenario seed data.
Args:
test_run_id: The UUID of the test run.
scenario_index: 1-based index of the conversation within
the test run. Visible in get_test_run_results output.
test_id: UUID of the specific data point. Alternative to
scenario_index.
| Name | Required | Description | Default |
|---|---|---|---|
| test_id | No | ||
| test_run_id | Yes | ||
| scenario_index | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true and idempotentHint=true, so the tool's safety profile is known. The description adds value by enumerating the exact return contents (model_input, model_result, metric_value, scenario seed data) and noting it returns a 'complete' transcript. It does not contradict annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured and front-loaded: it opens with the main purpose, then gives usage context, then details the return values, and finally lists parameters in an Args block. Despite its length, every sentence serves a purpose and adds necessary detail that the schema omits.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simple retrieval nature, good annotations, and existing output schema, the description covers all necessary context: what it does, when to use it, how to select the target conversation, and what it returns. No additional information is needed for a competent agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, but the description fully compensates by explaining all three parameters: test_run_id as UUID, scenario_index as a 1-based index visible in get_test_run_results, and test_id as an alternative UUID. This goes beyond the bare schema, which lacks descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific action ('Retrieve the full conversation transcript') and a specific resource ('a single data point'). It clearly distinguishes itself from sibling tools like get_test_run_results by noting it is for drilling into a specific conversation after summary results.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly says 'Use this after get_test_run_results to drill into a specific conversation,' providing clear context. It also explains how to identify the conversation (scenario_index or test_id) but does not explicitly mention when not to use the tool or name alternative tools for other purposes.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_dashboardGet DashboardARead-onlyIdempotentInspect
Retrieve a dashboard's full configuration by name.
Args:
name: Name of the dashboard to retrieve.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, covering the safety profile. The description adds the 'full configuration' detail, indicating the complete payload is returned, but it does not discuss error scenarios, permissions, or other non-safety behaviors. With annotations present, this is adequate but not rich.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The main description is a single, front-loaded sentence that immediately conveys the purpose. The Args block is redundant with the schema but is brief and does not add significant clutter. Overall, it is concise and well-structured.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity (one parameter, read-only, idempotent) and the existence of an output schema, the description is largely sufficient. It clearly states what it retrieves and how. A mention of when to prefer this over list_dashboards would improve completeness, but the core information is present.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema provides no parameter description (0% coverage), so the description must compensate. It does so by clarifying that the 'name' parameter is the dashboard's name, but this is only a minimal restatement of the parameter's title. It adds no detail about format, uniqueness, or validation, which is acceptable for a simple getter but does not go beyond the obvious.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action (retrieve), the resource (a dashboard's full configuration), and the qualifier (by name). This distinguishes it from sibling tools like list_dashboards, which lists dashboards, and save_dashboard/delete_dashboard, which modify them.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description gives clear context for when to use the tool: retrieve a specific dashboard's full configuration by name. It does not explicitly mention alternatives or exclusions, but the purpose is specific enough that the usage scenario is unambiguous.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_docsQuery Okareo DocumentationARead-onlyIdempotentInspect
Query the Okareo documentation system for information about Okareo primitives and workflows.
Use this tool when the agent or user needs to understand how Okareo
concepts work — Scenarios, Checks, Targets, Drivers, Evaluations,
and Simulations.
Two modes are available:
- 'conceptual': Detailed technical documentation for agent reasoning.
Default top_k=5 (returns up to 5 documentation entries).
- 'user_legible': Plain-language explanations for human users.
Default top_k=3 (returns up to 3 documentation entries).
If the Okareo documentation service is unavailable (e.g. air-gapped
environment), the tool returns a helpful error suggesting get_templates
as a fallback.
Args:
query: The question to ask the Okareo documentation system. Be
specific — e.g., 'How do Checks and Evaluations work together?'
or 'What is a Driver persona?'
mode: Documentation mode — 'conceptual' or 'user_legible'.
top_k: Number of documentation entries to return. Defaults to 5
for conceptual mode, 3 for user_legible mode. Maximum 10.
| Name | Required | Description | Default |
|---|---|---|---|
| mode | Yes | ||
| query | Yes | ||
| top_k | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description goes well beyond the annotations (readOnlyHint, openWorldHint, idempotentHint, destructiveHint) by disclosing mode-specific behaviors: 'conceptual' default top_k=5, 'user_legible' default top_k=3, maximum top_k=10, and a helpful error message suggesting get_templates as a fallback. These details describe the tool's dynamic behavior and error handling, which are not inferable from annotations alone.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured and front-loaded: first sentence states purpose, followed by usage guidance, mode details, error behavior, and parameter explanations. While longer than average, every sentence adds value—no redundancy. The use of bullet points and an Args section makes it scannable and easy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Despite having multiple parameters and no schema descriptions, the description is complete for a read-only documentation query tool. It covers all parameter semantics, mode behavior, defaults, and fallback scenarios. The presence of an output schema means return-value details are not required, so the description is sufficient for an agent to correctly select and invoke the tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema provides only parameter names and types (query, mode, top_k, with top_k nullable and default null). The description compensates fully by explaining each parameter: query with concrete examples ('How do Checks and Evaluations work together?'), mode with its two possible values ('conceptual' or 'user_legible'), and top_k with its mode-specific defaults and maximum value (10). This is essential since the schema has no descriptions (0% coverage).
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Query the Okareo documentation system for information about Okareo primitives and workflows.' It uses a specific verb (query) and resource (documentation system), and explicitly lists the concepts covered (Scenarios, Checks, Targets, Drivers, Evaluations, Simulations). This also distinguishes it from sibling tools like get_templates, which is mentioned as a fallback.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicit guidance is provided: 'Use this tool when the agent or user needs to understand how Okareo concepts work.' It also suggests a specific alternative (get_templates) in the event the documentation service is unavailable. The description clearly indicates when to use this tool and provides a fallback option, making it easy to choose among siblings.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_driverGet DriverARead-onlyIdempotentInspect
Retrieve a driver persona you've already configured.
Retrieves a Driver by name, returning its full configuration including the
persona prompt.
Args:
name: Name of the driver to retrieve.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare this as read-only and idempotent. The description adds that the full configuration including persona prompt is returned, which is useful context. No mention of error behavior, but given the simple read-only nature, this is acceptable and adds some value beyond annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact and front-loaded with the primary purpose. The additional sentence and Args section add value without redundancy or unnecessary detail.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple getter with one parameter, an output schema, and read-only annotations, the description covers the core behavior and return content. It doesn't discuss error handling, but the output schema and simplicity mitigate the gap, making the tool adequately complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has a single required string parameter 'name' with no description. The description's Args section restates that name is the driver name, providing minimal additional meaning. It doesn't add details like uniqueness or format, but for a simple identifier it suffices given the low schema coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool retrieves an existing driver persona by name, returning its full configuration including the persona prompt. This distinguishes it from sibling list_drivers, which lists drivers, and create_or_update_driver, which creates/updates.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for fetching a previously configured driver by name. It doesn't explicitly mention alternatives or when not to use it, but the context is clear enough: use when you need a specific driver's full config. It provides no exclusion rules but the intended use case is evident.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_generation_modelGet Generation ModelARead-onlyIdempotentInspect
Read detailed information about a registered generation model.
Returns the generation model's target LLM configuration, tags, creation
time, and any warnings (e.g., if the target LLM has been deprecated).
Args:
name: Name of the registered generation model.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false. The description adds useful behavioral context by listing exactly what is returned (target LLM config, tags, creation time, warnings) and explains what warnings may indicate (deprecation). No contradiction with annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and well-structured: an opening summary, a sentence on return values, and the Args breakdown. Every sentence provides necessary detail with no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple read operation with one parameter and an output schema, the description is complete. It covers the purpose, return content, and parameter semantics. No additional context is needed for correct invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema provides only a parameter name with no description (0% coverage). The description's Args section explicitly states 'name: Name of the registered generation model,' fully compensating for the schema's lack of detail. This adds clear meaning beyond the bare type.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action ('Read detailed information about a registered generation model') with a specific verb and resource. It distinguishes from sibling tools like list_generation_models by emphasizing detailed single-item retrieval.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use the tool (to read details of an existing generation model), but it does not explicitly mention alternatives or exclusion criteria. It does not say 'use list_generation_models to list all models' which would make it a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_reps_baselineGet REPS Baseline MaterialARead-onlyIdempotentInspect
Serve REPS agent-evaluation baseline material (scenario banks, drivers, checks, eval configs).
REPS is Okareo's agent-evaluation workbench: per-pillar baseline
material for evaluating AI agents across R-reasoning, E-execution,
P-performance, and S-security, plus shared explore/ probes and a
profile/ example. The material is published as tagged releases of
the okareo-tools repo; this tool serves the latest release so reps
skills need no local copy of the tree.
Two modes:
- Discovery (omit `path`): list what files exist in the served
release — the full tree, or one area via `pillar`. File lists
change between releases, so always discover before fetching.
- Fetch (provide `path`): return one file's exact content as
published in the release. Use paths verbatim from discovery,
e.g. 'S-security/scenarios/verification-gate.jsonl'.
Every response carries the release tag it was served from (e.g.
'v0.5.1') — record it in evaluation reports as baseline
provenance. `stale: true` means the last release check failed and
the content may lag the newest release.
Args:
pillar: Optional discovery filter. One of: R-reasoning,
E-execution, P-performance, S-security, explore, profile.
Omit to list the entire baseline tree (which also includes
shared material outside these areas).
path: Optional file path (relative to the baseline tree, as
returned by discovery). Provide to fetch that file's
content; omit for discovery.
version: Optional release tag. Currently only the served tag
is available; any other value returns an error naming what
IS available. Omit to accept the served release.
| Name | Required | Description | Default |
|---|---|---|---|
| path | No | ||
| pillar | No | ||
| version | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate read-only, idempotent, and non-destructive behavior. The description adds context about release provenance, staleness (`stale: true`), and the served-release-only version constraint, which are not inferable from annotations. It does not contradict annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is comprehensive but efficiently structured: overview, two modes, args, and response metadata. Each sentence adds distinct value, and the formatting makes it easy to parse. There is no redundancy or filler.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's dual-mode complexity, the description fully covers discovery and fetch behavior, parameter semantics, release provenance, and staleness warnings. The presence of an output schema means return-value details are not required in the description, and the description still notes key response fields.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has 0% parameter descriptions, so the description fully compensates. It thoroughly explains `pillar` (discovery filter with enumerated values), `path` (fetch mode, use verbatim from discovery), and `version` (served release only, error otherwise), including an example path.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb+resource: 'Serve REPS agent-evaluation baseline material' and defines REPS, clearly distinguishing it from sibling tools that manage individual scenarios or checks. It precisely states that the tool serves baseline files from tagged releases.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It explicitly explains two modes (discovery vs fetch), provides a rule to always discover before fetching, and describes when to use the `pillar` filter and `version` parameter. It also notes that any version other than the served tag returns an error naming available versions, giving clear expected behavior.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_scenarioGet ScenarioARead-onlyIdempotentInspect
Read a scenario's metadata and all data rows.
Look up by name or scenario ID. Returns scenario details and all
input/result data rows.
Args:
name: Name of the scenario to retrieve.
scenario_id: ID of the scenario to retrieve. Takes precedence over name.
| Name | Required | Description | Default |
|---|---|---|---|
| name | No | ||
| scenario_id | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The annotations already declare the tool safe (readOnlyHint=true, idempotentHint=true, destructiveHint=false). The description adds useful behavioral details: scenario_id takes precedence over name, and it returns all input/result data rows. It doesn't contradict annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is front-loaded with a one-sentence summary, followed by a structured Args block. There is slight redundancy between the first two sentences, but overall it's concise and well-organized.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple read-only tool, the description covers the lookup method, return content, and parameter precedence. The output schema handles return structure. However, it doesn't specify behavior when no arguments are provided or when a scenario is not found, which is a minor gap.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema description coverage, the description's Args section is the only source of parameter meaning. It clearly explains both 'name' and 'scenario_id', including the precedence rule, fully compensating for the empty schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Read') and identifies the resource ('a scenario's metadata and all data rows'), which clearly distinguishes it from sibling tools like list_scenarios or get_check. The purpose is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context: use this tool to read a scenario's metadata and data rows, with lookup by name or ID. It doesn't explicitly name alternatives or state when not to use it, but the read-only nature and specific resource make the use case clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_targetGet TargetARead-onlyIdempotentInspect
Check the current configuration of a test target.
Retrieves a Target by name. Works for all target types (Generation,
Custom Endpoint, and Voice).
For **custom_endpoint** Targets, the response is a flat envelope whose
keys mirror the kwargs accepted by `create_or_update_target`, so a
copilot can read the result, swap in a new name + secrets, and feed it
back to create to clone the Target. Fields the backend keeps secret
(those listed in `sensitive_fields`) appear with the literal value
`"***REDACTED***"` — these MUST be replaced with real values before
calling `create_or_update_target`, which rejects payloads still
containing the sentinel.
The `max_parallel_requests` field on custom_endpoint Targets is the
same setting the Okareo web UI labels "max concurrency".
For generation and voice Targets, the response shape is unchanged
(kept stable for existing callers).
Args:
name: Name of the target to retrieve.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Goes well beyond the readOnly/idempotent annotations by revealing specific behavioral details: sensitive fields appear as '***REDACTED***' and must be replaced before create_or_update_target, the max_parallel_requests field matches the UI label 'max concurrency', and the response shape remains stable for generation/voice targets. These are valuable insights not derivable from annotations alone.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with a clear opening statement, followed by detailed but relevant caveats and an Args section. Every sentence adds value and the length is justified by the multi-target-type nuances and redaction behavior.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers all target types, special handling for custom_endpoint, redacted fields, field naming mapping, and stability guarantees. With an output schema present, the description does not need to list return fields. It is highly complete for a read-only getter tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has no property description (0% coverage), so the description compensates by stating 'name: Name of the target to retrieve.' This adds straightforward meaning to the single required parameter. Though minimal, it is sufficient for a simple string parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function with a specific verb+resource: 'Check the current configuration of a test target' and 'Retrieves a Target by name.' It distinguishes itself from sibling tools like list_targets by indicating single-target retrieval by name and explicitly mentions support for all target types.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies when to use the tool (when needing a single target's configuration) and provides context for different target types, but it does not explicitly contrast with list_targets or state exclusions. The intended use is clear enough from the phrasing and sibling tool names.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_templatesGet Okareo TemplatesARead-onlyIdempotentInspect
Retrieve prompt templates for common Okareo patterns.
Returns starter templates for building Okareo test components. These
templates are served as static content from the MCP — no network calls
required. Always available, even in air-gapped environments.
Available templates:
- basic_scenario: Template for creating a basic Okareo test scenario
- boolean_check_prompt: Template for a pass/fail (boolean) check prompt
- score_check_prompt: Template for a scored check prompt
- check_code: Template for a code-based check (Python function)
- target_validate_check_prompt: Template for validating target output
- driver_prompt: Template for a Driver persona prompt
- driver_voice_extension_prompt: Template for voice interaction extensions
- analysis_check_prompt: Template for an analysis check (qualitative feedback)
Args:
template_name: Template identifier to retrieve. Omit to get a
lightweight listing of all available templates (names and
descriptions only). Provide a template_name to get the full
template content. Valid values: basic_scenario,
boolean_check_prompt, score_check_prompt, check_code,
target_validate_check_prompt, driver_prompt,
driver_voice_extension_prompt, analysis_check_prompt.
| Name | Required | Description | Default |
|---|---|---|---|
| template_name | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations (readOnlyHint, idempotentHint, destructiveHint), the description adds valuable behavioral context: templates are served as static content with no network calls, always available even in air-gapped environments. It also explains the differing behavior when the parameter is omitted vs provided, which is not captured by annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with an introductory sentence, a note on static availability, a bulleted list of templates, and an Args block. Despite being long, it is concise because every line adds necessary information. It is front-loaded with the primary purpose and does not contain fluff.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has a single optional parameter and an output schema. The description explains what the output will be in both modes (listing or full content) and lists all valid template names. Given the output schema exists, no further return-value detail is needed. The description is complete for the tool's simplicity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema provides zero description coverage for template_name (just a title and type). The tool description compensates fully by documenting the parameter, listing all valid values, and explaining the behavior when omitted. This is exemplary parameter documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: 'Retrieve prompt templates for common Okareo patterns.' It uses a specific verb (retrieve) and resource (prompt templates), and the list of available templates further distinguishes this tool from any sibling tool. No ambiguity about what the tool does.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear usage guidance: omit template_name to get a listing, or provide it to get full template content. It also explains the static, offline nature of the content. However, it does not explicitly compare to alternative tools or state when not to use it, though no sibling tool serves the same purpose.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_test_run_resultsGet Test Run ResultsARead-onlyIdempotentInspect
Load the results of a specific test run.
Look up by test run ID (UUID) or by name (returns the most recent run
matching that name). Returns aggregate metrics and per-row check scores.
By default, conversation transcripts (model_input/model_result) are
excluded to keep responses concise. Set include_transcripts=True to
include full transcripts. Use get_conversation_transcript to inspect
a single conversation's transcript without loading all of them.
Supports pagination via limit and offset for large result sets.
Args:
test_run_id: The UUID of the test run. Takes precedence over name.
name: The name of the test run. Returns the most recent match.
include_transcripts: Include full model_input and model_result in
each data point. Defaults to False (scores only). Set True for
full conversation transcripts.
limit: Maximum number of data points to return. 0 (default) returns
all data points. Use with offset for pagination.
offset: Number of data points to skip. Defaults to 0.
| Name | Required | Description | Default |
|---|---|---|---|
| name | No | ||
| limit | No | ||
| offset | No | ||
| test_run_id | No | ||
| include_transcripts | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already provide readOnly, idempotent, and non-destructive hints. The description adds valuable behavioral context beyond annotations: transcripts are excluded by default for conciseness, test_run_id takes precedence over name, and name returns the most recent match. It does not contradict annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with a summary paragraph and an Args list. It is somewhat lengthy but every sentence adds useful detail. The front-loaded purpose and logical flow make it easy to scan. Slight redundancy exists between the prose and Args list, but it aids clarity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The tool has an output schema, so return values are covered. The description explains lookup methods, alternative tool for transcripts, pagination, and default behavior. It is complete for a read-only query tool with 5 parameters; no crucial information is missing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage, so the description carries full burden. The Args section provides detailed semantics for all 5 parameters: precedence of test_run_id, behavior of name, effect of include_transcripts, and pagination semantics of limit/offset. This far exceeds what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool loads results of a specific test run, with specific lookup methods (by UUID or name) and what it returns (aggregate metrics and per-row check scores). It distinguishes itself from sibling tools like get_conversation_transcript and list_test_runs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly tells when to use this tool vs alternatives: 'Use get_conversation_transcript to inspect a single conversation's transcript without loading all of them.' It also explains when to use include_transcripts and pagination, giving clear contextual guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_voice_integrationGet Voice IntegrationARead-onlyIdempotentInspect
Retrieve a voice provider integration by id, including its status.
Args:
integration_id: The integration's id (from list_voice_integrations).
| Name | Required | Description | Default |
|---|---|---|---|
| integration_id | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is well covered. The description adds that the result includes 'status,' which is useful, but doesn't cover other behavioral aspects like pagination, error handling, or auth requirements. Given the strong annotation coverage, the additional context is sufficient but not rich.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is brief and front-loaded with the core purpose in the first sentence. The Args section is minimal and directly tied to the schema parameter. Every sentence earns its place without redundancy or fluff. This is an example of efficient, well-structured documentation.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple get-by-id tool with one parameter, strong annotations (read-only, idempotent), and an output schema, the description covers the necessary context: what to pass and that it returns status. The only missing element is explicit behavior for non-existent ids (e.g., not-found error), but given the tool's simplicity and existing structured metadata, this is adequate.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema provides only the parameter name and type (integration_id: string) with no description. The tool description compensates by explaining the parameter's origin and meaning: 'The integration's id (from list_voice_integrations).' This adds valuable semantic context beyond the schema, telling the agent where to obtain the id and implying it should reference a valid existing integration.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action: 'Retrieve a voice provider integration by id, including its status.' It uses a specific verb (retrieve), identifies the resource (voice provider integration), and differentiates from siblings like list_voice_integrations by specifying lookup by id. This leaves no ambiguity about what the tool does.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear context for when to use the tool: 'by id' and the parameter hint 'from list_voice_integrations' implies a prerequisite workflow—first list integrations to obtain the id, then retrieve details. It doesn't explicitly name alternatives or exclusions, but the usage context is clear enough for the intended purpose.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
get_voice_webhook_urlGet Voice Webhook URLARead-onlyIdempotentInspect
Get the inbound webhook endpoint for a voice provider.
Paste the returned URL into the provider's console so its call traffic
reaches Okareo monitoring.
Args:
provider: Voice platform — one of: retell, twilio, vapi, elevenlabs.
public_id: The integration's public_id (from connect_voice_integration
or get_voice_integration). Required for retell and twilio.
| Name | Required | Description | Default |
|---|---|---|---|
| provider | Yes | ||
| public_id | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, openWorldHint, idempotentHint, and destructiveHint:false. The description adds useful context beyond these annotations, such as the purpose of the URL (reaching Okareo monitoring) and the conditional necessity of public_id for certain providers. It doesn't describe return format or error behavior, but annotations cover safety/read-only aspects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and well-structured: a clear purpose sentence, a usage directive, and a neatly formatted Args list. Every sentence adds value without unnecessary verbosity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's low complexity, an output schema is present, and annotations are rich, the description adequately covers purpose, usage, and parameter semantics. It leaves no critical gaps for an agent to invoke the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has no descriptions (0% coverage), but the description's Args section comprehensively explains each parameter: provider lists valid enum values, and public_id is defined with its source and when it is required. This fully compensates for the schema's lack of detail.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description uses a specific verb ('Get') and resource ('inbound webhook endpoint for a voice provider'), clearly distinguishing it from sibling integration tools like connect_voice_integration or get_voice_integration. It also states the intended use of the returned URL, further clarifying its purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description clearly explains when to use the tool (to obtain a webhook URL for a voice provider) and what to do with the result (paste it into the provider's console). It notes provider-specific requirements for public_id, which is important context. However, it does not explicitly mention when not to use this tool or name alternative tools.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
ingest_conversationsIngest ConversationsAInspect
Submit completed voice conversations to Okareo for monitoring.
Each conversation's turns become evaluable data points and any
configured monitors run their checks automatically. Use this to feed
production voice traffic (Retell, Twilio, VAPI, ElevenLabs, or a custom
source) into Okareo monitoring.
Conversations are validated individually: valid ones are ingested and
invalid ones are returned in a "rejected" list — the batch is not
all-or-nothing.
Args:
conversations: List of conversation objects. Each MUST include a
"call_id" and at least one of: "transcript" (a list of
{role, content, timestamp_ms} turns), "audio"
({"type": "url"|"voice_file_id"|"inline_b64", ...}),
"recording_url", or "recording_bytes_b64". Optional per
conversation: "context_token", "metadata", "tags" (tags drive
monitor/filter-group matching), "diarization", "first_turn".
When both a transcript and audio are supplied, the transcript
takes precedence.
project_id: Okareo project ID. Defaults to the account's project.
mut_id: Optional model-under-test ID. Omit for pure monitoring —
data points are then matched to monitors by their tags only.
| Name | Required | Description | Default |
|---|---|---|---|
| mut_id | No | ||
| project_id | No | ||
| conversations | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Even with annotations present (readOnlyHint=false, openWorldHint=true), the description adds substantial behavioral context: turns become evaluable data points, monitors run automatically, validation is per-conversation (not all-or-nothing), rejected list is returned, and transcript precedence when both transcript and audio are provided. This goes beyond what annotations reveal.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is structured effectively: a clear opening sentence, a usage paragraph, a validation behavior sentence, and a well-organized Args list. Every sentence carries essential information for correct invocation, and the length is justified by the complexity of the conversation object structure. No fluff or redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (nested conversation objects, multiple optional fields, integration with monitors, validation semantics) and the presence of an output schema, the description covers all necessary aspects: purpose, use cases, parameter details, validation behavior, and tag/mut_id matching. It leaves no significant ambiguity for an agent to invoke the tool correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, but the description fully compensates with a detailed Args section. It explains the conversations array's required fields (call_id, at least one of transcript/audio/recording_url/recording_bytes_b64), optional fields, and the exact semantics of project_id and mut_id, including the default and tagging behavior. This is exemplary compensation for missing schema descriptions.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb+resource+purpose: 'Submit completed voice conversations to Okareo for monitoring.' It clearly distinguishes this from sibling tools like run_simulation and run_test by focusing on production voice traffic ingestion. The scope is unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit context: 'Use this to feed production voice traffic (Retell, Twilio, VAPI, ElevenLabs, or a custom source) into Okareo monitoring.' It also clarifies behavior like per-conversation validation and mut_id omission for pure monitoring. However, it does not explicitly name alternatives or when-not-to-use conditions, so it falls short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_available_llmsList Available LLMsARead-onlyIdempotentInspect
Browse available LLMs from the Okareo registry.
Returns all LLMs that can be used when registering a generation model
for testing. Each entry has a name, display name, and provider. Use a
model_name from this list when calling register_generation_model.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, and non-destructive behavior. The description adds useful context by stating the data source (Okareo registry) and the shape of entries (name, display name, provider). This goes beyond the basic safety profile and explains how the output should be consumed.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three sentences, front-loaded with the core purpose, and every sentence earns its place. It avoids fluff and directly states what, why, and how to use the result. Excellent conciseness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a zero-parameter listing tool with strong annotations and an output schema, the description is complete. It covers the registry source, the purpose (registration), and the result fields. No additional behavioral or usage context is needed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so the input schema is trivial. Per rubric, a baseline of 4 is appropriate. The description adds no parameter details (none exist), but it does explain the output usage, which is the only semantically relevant information needed.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function: 'Browse available LLMs from the Okareo registry' and 'Returns all LLMs that can be used when registering a generation model.' It distinguishes this from sibling tools like list_generation_models by explicitly tying it to the registry and the register_generation_model workflow.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit guidance: 'Use a model_name from this list when calling register_generation_model.' This tells the agent exactly when to use this tool. It doesn't discuss when not to use it or contrast with alternatives, but the usage context is clear.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_checksList ChecksARead-onlyIdempotentInspect
List available quality checks, grouped by category.
Returns checks (both built-in and custom) available in your Okareo
account, organized into `checks_by_category` using the platform's
`__category:<Category>` tags; checks with no category appear under
`uncategorized`. Select checks from the category matching your task
AND modality: voice-specific categories (e.g. voice/audio quality)
apply to voice simulations, while checks outside voice-specific
categories are generally useful for both chat and voice. A check
carrying multiple categories appears under each of them.
Each check has a name, description, and output_data_type.
output_data_type uses the server vocabulary: "bool" is a pass/fail
check and "int" is a scored check — these correspond to output_type
"pass_fail" and "score" in create_or_update_check and generate_check.
Use these check names with run_test to evaluate model quality.
Args:
limit: Maximum number of checks to return (default 20), applied to
the total before grouping. Use 0 for no limit.
all_versions: When false (default), returns only the latest version of
each check. When true, returns the full version history of every
check, each entry annotated with its version number.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| all_versions | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the read-only and idempotent annotations, the description reveals key behavioral details: grouping by `__category:<Category>` tags, handling of uncategorized checks, duplicate entries for multi-category checks, the meaning of output_data_type values ('bool', 'int') and their mapping to output_type, and version behavior controlled by all_versions. This adds significant value beyond the structured annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but well-structured with clear paragraphs and an Args section. The first sentence captures the core purpose. Each subsequent sentence adds necessary context about grouping, output semantics, and parameters. It is appropriately verbose for the tool's complexity.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present, the description doesn't need to detail return fields, but it still explains the `checks_by_category` organization, uncategorized fallback, and output_data_type semantics. It also covers edge cases like multi-category checks and version history. The tool is fully specified for an agent.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has no property descriptions, so the description carries the full burden. The Args section thoroughly explains `limit` (default, semantics of 0, application before grouping) and `all_versions` (latest vs full history, version annotation). This exceeds what the schema provides.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'List available quality checks, grouped by category', clearly identifying the action (list) and resource (quality checks). It distinguishes itself from sibling tools focused on creating, updating, or deleting checks.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear guidance on how to choose checks: 'Select checks from the category matching your task AND modality' and explains that voice-specific categories apply to voice simulations. It also directs the user to use check names with run_test, establishing the tool's role in the evaluation workflow. While it doesn't explicitly contrast with sibling tools, the context is sufficient.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_dashboardsList DashboardsARead-onlyIdempotentInspect
List the analytics dashboards in your Okareo project.
Args:
limit: Maximum number of dashboards to return (default 20). Use 0
for no limit.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false. The description adds useful behavioral context about the limit parameter, specifically that 0 means no limit, which is not in the schema and informs the agent of a special behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two sentences and front-loaded with the action. The Args block is compact and directly relevant. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple list tool with one parameter, the description covers purpose and parameter behavior. The output schema covers return structure, so no need to describe it. It could mention pagination or ordering, but that isn't required for straightforward listing.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema only provides the parameter type and default (20), while the description explains that limit is the maximum number of dashboards to return and that 0 disables the limit. This adds meaningful semantics beyond the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the verb 'List' and the resource 'analytics dashboards in your Okareo project.' It distinguishes from siblings like get_dashboard (which fetches a single dashboard) and other list_* tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The usage is implied by the name and description: use to list dashboards. However, there is no explicit guidance on when to prefer this over get_dashboard or other list tools, nor any mention of alternatives.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_driversList DriversARead-onlyIdempotentInspect
See what driver personas are available in this project.
Returns all Drivers with their names, IDs, model, and temperature.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate a safe read-only, idempotent operation. The description adds value by disclosing exactly what is returned (names, IDs, model, temperature) and that it returns all drivers, which goes beyond the minimal annotation information.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is two short sentences, front-loaded with 'See what driver personas are available' and immediately followed by the return fields. No wasted words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple list tool with no parameters, an output schema, and a clear description of the returned data, this is fully complete. The output schema handles detailed return shape, and the description provides the semantic content.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so there is nothing to explain. The description correctly avoids any parameter details, and the baseline for zero-parameter tools is 4.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool lists all driver personas with specific fields (names, IDs, model, temperature). This distinguishes it from related tools like get_driver (single driver) and list_driver_voices (voices only).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The context is clear: use this when you need an overview of all available drivers. It does not explicitly mention alternatives or exclusions, but the simplicity and scope ('all Drivers') make the use case obvious without extra guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_driver_voicesList Driver VoicesARead-onlyIdempotentInspect
Discover the voices, voice profiles, and languages available for configuring voice-capable drivers.
Call this before create_or_update_driver when building a voice agent
simulation, so you can pass valid `voice`, `voice_profile`, and
`language` values.
Each entry in `voices` carries selection metadata — use it to pick
the voice:
- `language`: bare ISO code (e.g. "en", "es", "ja"). The driver's
`language` is derived from the selected voice's language when
omitted, and must not conflict with it.
- `accent`: free-text accent label present on many voices (e.g.
"British", "Mexican", "Parisian", "Southern US"). To satisfy an
accent request, select a voice whose `accent` matches — writing
accent instructions into `voice_instructions` does NOT change the
TTS voice.
- `gender`: e.g. "feminine" / "masculine".
`voice_profiles` are emotion/delivery presets (happy, angry,
sarcastic, ...) — they shape affect, not accent or language.
`languages` lists the distinct voice languages available.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, and non-destructive, so the description need not repeat these. It adds valuable context about the data semantics (e.g., `language` derivation and conflict rule, `accent` matching requirement) and clarifies the limitation of `voice_instructions`, going beyond the structured annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is longer than typical, but it earns its length with structured bullet points and front-loaded purpose/usage statements. Every sentence adds meaningful guidance, and the format aids readability despite the density.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has no parameters and an output schema exists, the description is fully complete for an agent to invoke it correctly. It covers the key concepts (voices, profiles, languages), selection criteria, and integration with create_or_update_driver, leaving no critical gaps.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so the baseline is 4. The description compensates by thoroughly explaining the output fields (language, accent, gender) and their intended use, which is more relevant than parameter docs for this read-only list operation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb+resource+scope: 'Discover the voices, voice profiles, and languages available for configuring voice-capable drivers.' It clearly distinguishes the tool from sibling list tools by focusing on voice configuration and explicitly references create_or_update_driver, making its purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit when-to-use guidance: 'Call this before create_or_update_driver when building a voice agent simulation, so you can pass valid `voice`, `voice_profile`, and `language` values.' It also offers selection guidelines and a direct exclusion ('writing accent instructions into `voice_instructions` does NOT change the TTS voice'), effectively preventing misuse.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_generation_modelsList Generation ModelsARead-onlyIdempotentInspect
Browse all registered generation models in the project.
Returns generation model names, IDs, target LLM configurations, and creation timestamps. Use this to see what generation models are available for testing.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, establishing a safe read operation. The description adds scope ('registered', 'project') and return fields, providing modest extra context beyond annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is three short sentences, front-loaded with the action and resource. Every sentence contributes value: the action, the returned data, and the intended use case.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With no parameters and an output schema present, the description sufficiently covers the purpose and return content. The mention of 'registered models in the project' clarifies scope, and the use case adds practical context.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, and schema coverage is trivially 100%. The description does not need to explain parameters; the baseline for 0 params is 4, and no additional param information is required.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool lists all registered generation models, specifying returned fields (names, IDs, target LLM configurations, creation timestamps). This distinguishes it from siblings like get_generation_model (single model) and list_available_llms (available LLMs).
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides a clear use case: 'Use this to see what generation models are available for testing.' It does not explicitly mention alternatives or when not to use this tool, but the context is sufficient.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_scenariosList ScenariosARead-onlyIdempotentInspect
List scenarios in the project, most recent first.
Returns scenario names, IDs, tags, row counts, and creation dates.
Use this to discover existing scenarios before running a test.
Args:
limit: Maximum number of scenarios to return (default 20).
Set to 0 to return all scenarios.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare this as read-only, idempotent, and non-destructive. The description adds meaningful behavioral context beyond annotations, such as 'most recent first' ordering and the semantics of the limit parameter (default 20, 0 returns all). No contradictions exist.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and well-structured: main behavior and ordering in the first sentence, returned fields in the second, usage guidance in the third, and a clean Args section. Every sentence adds value, and the format is easy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
The description fully covers the single optional parameter and the tool's purpose. With an output schema present, it doesn't need to detail return structure. It could mention the project context more explicitly, but the description is sufficiently complete for a list tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema only provides type and default for the limit parameter, but the description fully explains its meaning: 'Maximum number of scenarios to return (default 20). Set to 0 to return all scenarios.' This goes well beyond the schema's minimal coverage.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool lists scenarios in the project, most recent first, and enumerates the returned fields (names, IDs, tags, row counts, creation dates). This specific verb+resource combination distinguishes it from siblings like get_scenario or save_scenario.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides a clear context: 'Use this to discover existing scenarios before running a test.' While it doesn't explicitly mention alternatives or when-not-to-use, the guidance is sufficient for a list operation, and no other sibling offers the same discovery functionality.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_simulationsList SimulationsARead-onlyIdempotentInspect
List past simulation runs in the project.
Returns simulation run names, IDs, timestamps, and status, sorted by
most recent first. Defaults to the 10 most recent runs in summary mode.
Use detail_level="detailed" to include model_metrics and additional
fields (limit is capped to 5 in detailed mode to prevent overflow).
Use get_test_run_results with the returned test_run_id to retrieve
per-row scores (transcripts excluded by default). Then use
get_conversation_transcript with a scenario_index to inspect
individual conversation transcripts.
Args:
target_name: Optional filter — only show simulation runs using
this target.
scenario_name: Optional filter — only show simulation runs using
this scenario.
limit: Maximum number of runs to return, sorted by most recent
first. Defaults to 10. Set to 0 to return all runs.
detail_level: "summary" (default) returns compact results without
model_metrics. "detailed" returns full results with metrics
(limit capped to 5).
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| target_name | No | ||
| detail_level | No | summary | |
| scenario_name | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description adds behavioral details beyond annotations, including sorting order ('sorted by most recent first'), default limit of 10, the cap at 5 for detailed mode, and the distinction between summary and detailed outputs. These are not conveyed by the readOnly/idempotent annotations, making the description valuable.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is well-structured with a clear summary, a usage workflow paragraph, and a labeled Args section. It is relatively detailed but each sentence serves a purpose, and the use of a code block for parameters improves readability.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the presence of an output schema (which covers return structure), the description covers the tool's purpose, all parameters, behavioral constraints, and its place in the workflow with sibling tools. It is complete enough for an agent to select and invoke the tool correctly without additional information.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema has no parameter descriptions (0% coverage), but the description thoroughly explains each parameter: target_name and scenario_name as optional filters, limit with its default and 0-to-return-all behavior, and detail_level with its two allowed values and the 5-cap implication. This fully compensates for the schema's lack of documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's function with a specific verb and resource ('List past simulation runs in the project'), and details what is returned (names, IDs, timestamps, status, sorted most recent first). It differentiates from sibling tools like get_test_run_results and get_conversation_transcript by outlining the intended workflow.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It provides explicit guidance on when to use this tool versus downstream tools, instructing to use get_test_run_results with the returned test_run_id and get_conversation_transcript with scenario_index. It also explains when to switch detail_level based on whether model_metrics are needed. This contextualizes usage within the broader API workflow.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_targetsList TargetsARead-onlyIdempotentInspect
Browse all simulation targets available in this project.
Returns all simulation targets (voice and custom_endpoint types)
created via create_or_update_target. Does not include generation models
registered via register_generation_model — use list_generation_models
for those.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already cover read-only and non-destructive nature. Description adds that it returns only voice and custom_endpoint types created via create_or_update_target, and excludes generation models—useful scope context that complements the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Description is short, front-loaded with the main purpose, and every sentence provides value without redundancy.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple list operation with an output schema, the description covers the essential scope, exclusions, and fallback for related entities. No gaps identified.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
No parameters, so schema is empty. Description adds meaning by clarifying the implicit selection criteria (types and origin), exceeding the baseline for zero-parameter tools.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Clearly states the tool lists all simulation targets, distinguishing from generation models by naming the alternative list_generation_models. The verb 'browse' is less specific than 'list' but resource and scope are explicit.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly instructs to use list_generation_models for generation models, providing a clear when-not-to-use and an alternative. Also scopes to the current project.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_tenantsList TenantsARead-onlyIdempotentInspect
List every Okareo organization you have access to in this MCP session.
The currently-active organization is marked ``is_current: true``. The
active organization is determined at sign-in (the token this session
presents is already scoped to it). Only available on OAuth-authenticated
sessions; on Bearer-API-key sessions returns
``tenant_selection_requires_oauth``.
| Name | Required | Description | Default |
|---|---|---|---|
No parameters | |||
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly/idempotent/destructive hints, and the description adds meaningful context: the is_current marker and the OAuth-only constraint, going beyond annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Three sentences, each providing distinct value: scope, active marker, and authentication requirement. No fluff, well organized.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the output schema exists and the tool has no parameters, the description fully covers behavior, including edge cases like non-OAuth sessions.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The tool has zero parameters, so the description need not explain semantics. The description implicitly confirms no inputs are needed.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description states a specific verb ('List') and resource ('every Okareo organization you have access to'), and distinguishes itself from siblings like switch_tenant by focusing on listing rather than switching.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly states when the tool is available (OAuth-authenticated sessions) and what happens on Bearer-API-key sessions, giving clear usage context and an alternative behavior.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_test_runsList Test RunsARead-onlyIdempotentInspect
List past test runs in the project.
Returns test run names, IDs, timestamps, status, and summary scores,
sorted by most recent first. Defaults to the 10 most recent runs.
Optionally filter by model name, scenario name, or type.
For simulation runs (type MULTI_TURN), use get_test_run_results with the
returned test_run_id to retrieve full conversation transcripts and per-turn
check scores.
Args:
model_name: Optional filter — only show test runs using this model.
scenario_name: Optional filter — only show test runs using this scenario.
limit: Maximum number of runs to return, sorted by most recent first.
Defaults to 10. Set to 0 to return all runs.
simulation_only: When True, return only MULTI_TURN simulation runs.
Useful for browsing past simulation results without NL_GENERATION or
other test run types appearing in the list.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No | ||
| model_name | No | ||
| scenario_name | No | ||
| simulation_only | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnly, idempotent, non-destructive; description adds default limit=10, limit=0 returns all, and that results are sorted by most recent. Also clarifies that simulation_only filters to MULTI_TURN type.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is organized with a clear opening sentence, followed by return values, defaults, filters, and a pointer to an alternative tool. The Args section is structured and each line adds value. No wordiness.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With a rich output schema present, the description need not detail return types, but it already summarizes return fields. It covers filters, defaults, and the recommended next step for transcripts. Sufficient for an agent to use correctly.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has zero field descriptions, but the description documents every parameter (model_name, scenario_name, limit, simulation_only) with defaults and purpose, fully compensating. The simulation_only explanation gives extra context on what MULTI_TURN simulation runs are.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
States it lists past test runs, specifies returned fields (names, IDs, timestamps, status, summary scores) and sort order. Clearly differentiates from get_test_run_results by directing users there for transcripts.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly directs users to get_test_run_results for simulation run details, implying this tool is for summary listing. Explains filtering options with examples of when simulation_only is useful.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
list_voice_integrationsList Voice IntegrationsARead-onlyIdempotentInspect
List the voice provider integrations in your Okareo project.
Args:
limit: Maximum number of integrations to return (default 20). Use 0
for no limit.
| Name | Required | Description | Default |
|---|---|---|---|
| limit | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, openWorldHint=true, idempotentHint=true, destructiveHint=false, so the safety profile is covered. Description adds the limit semantics (default 20, 0 for no limit), which is useful but does not disclose other behaviors like pagination or ordering.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
Description is short, front-loaded with a clear sentence, and an organized Args section. No redundant words.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a single-parameter, read-only list tool with an output schema, the description covers the core purpose and parameter semantics. The scoped context and limit behavior are sufficient.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The description explicitly explains the limit parameter, including the default value and the special meaning of 0 (no limit), which goes beyond the schema's default. This compensates for the low schema coverage, even though there is only one parameter.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states the action (list), resource (voice provider integrations), and scope (in Okareo project), distinguishing it from sibling tools such as get_voice_integration and connect_voice_integration.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance on when to use versus alternatives, such as get_voice_integration for a single integration. Usage is implied by the list operation, but no exclusions or use-case contexts are provided.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
preview_delete_scenarioPreview Scenario DeletionARead-onlyIdempotentInspect
Preview what will be deleted before removing a scenario.
Shows the scenario details and count of related test runs that will
also be deleted. Use delete_scenario to confirm deletion after reviewing.
Args:
name: Name of the scenario to preview deletion for.
scenario_id: ID of the scenario. Takes precedence over name.
| Name | Required | Description | Default |
|---|---|---|---|
| name | No | ||
| scenario_id | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is covered. The description adds valuable behavioral context beyond the annotations: it discloses that the tool shows scenario details and the count of related test runs, and that scenario_id takes precedence over name. This goes beyond what annotations provide, though it could mention edge cases like missing both parameters.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and well-structured, with the main purpose stated in the first sentence and an explicitly labeled Args block. Every sentence contributes meaningful information without fluff or redundancy. The front-loaded design aids quick comprehension.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's simplicity, the presence of an output schema (which covers return values), and the strong annotations, the description is complete for an agent to select and invoke the tool correctly. It covers the tool's purpose, the follow-up action, and the parameter nuances, making it fully adequate for the task.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema provides no descriptions for the two optional parameters (0% coverage), so the description carries this burden. It explains the meaning of 'name' and 'scenario_id' and the precedence relationship, which is essential for correct usage. However, it does not explicitly state that at least one parameter should be provided, leaving a minor ambiguity.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Preview what will be deleted before removing a scenario.' It specifies the action (preview) and the resource (scenario deletion), and it distinguishes itself from the sibling tool delete_scenario by explicitly naming the confirmation workflow. It also details what will be shown: scenario details and the count of related test runs.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit usage guidance: 'Use delete_scenario to confirm deletion after reviewing.' This tells the agent when to use this tool versus the alternative, making the workflow clear. The contextual hint that this is a pre-deletion review step is sufficient for tool selection.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
query_analyticsQuery AnalyticsARead-onlyIdempotentInspect
Query Okareo's product analytics to understand evaluation trends.
Answers questions like "how is my evaluation quality trending" by
aggregating measures across dimensions over a time window.
Args:
measures: Metrics to aggregate. Required. For the ``check_trend``
cube: avg_check_value, issue_rate, error_rate, datapoint_count,
issue_count, error_count, test_run_count, avg_latency, sum_cost,
input_token_count, output_token_count.
dimensions: Optional group-by fields (e.g. ["check.name"],
["target.name"], ["provider"]).
cube: Optional analytics cube name (defaults to ``check_trend``,
currently the only cube).
filters: Optional list of filter objects
``{"member": ..., "operator": ..., "values": [...]}``.
time_range: Optional look-back window — one of LAST_HOUR,
LAST_24_HOURS, LAST_7_DAYS, LAST_14_DAYS, LAST_30_DAYS,
LAST_90_DAYS. If neither time_range nor time_dimensions is
given, defaults to LAST_30_DAYS (the analytics API requires a
time window).
time_dimensions: Optional time bucketing — a list with at most one
entry, e.g. [{"dimension": "test_run.start_time",
"granularity": "day"}] (granularity: hour, day, or week).
include_metadata: When true, also return the available cubes,
dimensions, and measures so the query can be refined.
| Name | Required | Description | Default |
|---|---|---|---|
| cube | No | ||
| filters | No | ||
| measures | Yes | ||
| dimensions | No | ||
| time_range | No | ||
| time_dimensions | No | ||
| include_metadata | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark it as read-only and idempotent. The description adds useful behavioral context: the API requires a time window and defaults to LAST_30_DAYS, and include_metadata returns available cubes/dimensions/measures. This goes beyond the annotations without contradicting them.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is lengthy but every sentence adds value, especially given the 7 parameters. The structured Args list improves scannability. Slight verbosity in the intro sentence could be trimmed, but it's well-organized.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a complex tool with 7 parameters and an output schema, the description covers all necessary context: defaults, constraints, allowed values, and metadata option. The output schema exists, so return-value details are not required. No gaps identified.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, but the Args section fully documents each parameter: valid measure names, dimension examples, cube options, filter object format, time_range enum, time_dimensions structure with granularity, and include_metadata behavior. This is exceptional compensation for the schema.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb and resource: 'Query Okareo's product analytics to understand evaluation trends.' It clearly differentiates from sibling tools like get_check or list_test_runs by focusing on aggregation and trend analysis. The example question reinforces the purpose.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
It gives a concrete use case ('how is my evaluation quality trending') and notes the time-window requirement. It doesn't explicitly name alternative tools, but the context makes when to use this tool clear. Sibling names are mostly CRUD operations, so the differentiation is implicit.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
reevaluate_test_runRe-evaluate Test RunAInspect
Re-score a completed test run against a set of checks.
Re-runs checks against an already-finished test run without re-executing
the original model or simulation, and without changing the original
run's results. Useful after a check definition changed, or to score an
existing run against additional checks.
Args:
test_run_id: UUID or name of a completed test run.
checks: Optional list of check names (or IDs) to score against.
When omitted, the run's existing checks are re-run.
| Name | Required | Description | Default |
|---|---|---|---|
| checks | No | ||
| test_run_id | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Discloses that it does not re-execute the original simulation and does not change original run results, adding valuable context beyond the annotations. It does not mention idempotency or other side effects, but the output schema covers return behavior, and the annotation openWorldHint already signals possible side effects.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise, front-loaded with the main purpose, and includes a structured Args list. Every sentence adds value, making it easy to scan and understand quickly.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
With an output schema present, the description does not need to explain return values. Prerequisites, use cases, and parameter semantics are all clearly covered, so the tool can be invoked correctly without further information.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema has zero parameter descriptions, yet the description fully explains both parameters: test_run_id is a UUID/name of a completed run, and checks is an optional list with clear behavior when omitted. This completely compensates for the schema gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
Description clearly states it re-scores a completed test run without re-executing the model/simulation, distinguishing it from run_test and get_test_run_results. The verb 're-score' plus the qualifier 'without re-executing' provides precise scope and uniqueness.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Explicitly notes it is useful after a check definition changed or to score against additional checks, giving clear when-to-use context. However, it does not explicitly name alternative tools or state when not to use this tool, so it stops short of full guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
register_generation_modelRegister Generation ModelAIdempotentInspect
Register a generation model for testing by selecting an LLM from the registry.
Creates a generation model (Model Under Test) that points to a specific LLM
(e.g., 'azure/gpt-4o-mini'). Use list_available_llms to see available
LLMs. The registered generation model can then be used with run_test.
Args:
name: A human-readable name for this generation model (e.g., 'my-chatbot').
model_name: The LLM from the registry (from list_available_llms).
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | ||
| model_name | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations indicate idempotentHint=true, which is helpful, and the description adds that this 'creates a generation model (Model Under Test)' pointing to a specific LLM. It clarifies the registration semantics (registering creates an instance) and the relationship to testing, which goes beyond the annotations. It doesn't mention persistence or side effects, but the idempotentHint covers re-registration safety.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and well-structured. It front-loads the core purpose, provides usage guidance, and ends with parameter explanations. Every sentence earns its place, and the formatting with Args clearly separates parameter descriptions.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's moderate complexity, the description fully covers what an agent needs: the purpose, the required parameters, how to discover valid model_name values, and how the result is used. The presence of an output schema means the description doesn't need to explain return values. The sibling context (list_available_llms, run_test) is addressed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema coverage is 0%, so the description carries the full burden. It explains 'name' as 'a human-readable name' and 'model_name' as 'the LLM from the registry (from list_available_llms)', which adds meaning beyond the raw schema types. It could provide more detail on name uniqueness, but the description effectively maps parameters to their semantics.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's purpose: 'Register a generation model for testing by selecting an LLM from the registry.' It uses specific verbs ('register', 'creates') and identifies the resource ('generation model') and distinguishes it from sibling tools like list_generation_models or update_generation_model. The example LLM ('azure/gpt-4o-mini') further clarifies the intended use.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly tells users to use 'list_available_llms' to see available LLMs and mentions that the registered model can be used with 'run_test'. This provides clear when-to-use context and names alternatives/prerequisites, which is excellent guidance for an AI agent.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
reorder_dashboardsReorder DashboardsBIdempotentInspect
Set the display order of dashboards.
Args:
ordered_names: Dashboard names in the desired order.
| Name | Required | Description | Default |
|---|---|---|---|
| ordered_names | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already disclose readOnlyHint=false, destructiveHint=false, and idempotentHint=true, so the description doesn't need to repeat those. However, it adds no additional behavioral context, such as whether the order is fully replaced or how unlisted dashboards are handled. The description provides no new behavioral insight beyond the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise: a single sentence stating the purpose followed by a brief parameter definition. It is front-loaded and avoids unnecessary wording. The only minor issue is the embedded 'Args:' block format, but it remains efficient and easy to parse.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple reorder operation with one parameter and an output schema, the description provides the core information needed. However, it omits edge-case behavior, such as whether all dashboards must be included or how partial lists are handled. The tool is straightforward, but the description could be more complete with a note on ordering semantics.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With schema description coverage at 0%, the description compensates by explicitly defining the parameter: 'ordered_names: Dashboard names in the desired order.' This adds meaningful context beyond the schema's bare 'array of strings' type. It clearly explains what the parameter represents and how to use it.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the tool's action: 'Set the display order of dashboards.' This uses a specific verb and resource, distinguishing it from sibling tools like list_dashboards, save_dashboard, or delete_dashboard. The purpose is immediately clear and unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides no explicit guidance on when to use this tool versus alternatives. It only states the action without explaining prerequisites (e.g., dashboards must exist) or how it relates to other dashboard operations. The usage context is heavily implied rather than stated.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
rotate_voice_integration_secretRotate Voice Integration SecretADestructiveInspect
Rotate a voice provider integration's secrets.
Args:
integration_id: The integration's id.
secrets: The new provider secret values. The response returns only
a secret summary, never raw secret values.
| Name | Required | Description | Default |
|---|---|---|---|
| secrets | Yes | ||
| integration_id | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate destructive and non-read-only behavior. The description adds valuable context by stating 'The response returns only a secret summary, never raw secret values,' which goes beyond the annotations and informs the agent about the response behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise: one main sentence plus an inline parameter list. Every word earns its place, and the main action is front-loaded.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool has only two parameters, a nested object, and an output schema, the description covers the essential intent and the notable response behavior. It could mention prerequisites or side effects of rotation, but the existing annotations and the explicit note on response summaries make it fairly complete.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema description coverage, the description provides basic meaning for both parameters: integration_id is 'the integration's id' and secrets are 'the new provider secret values.' This is helpful but lacks detail on format, required structure, or constraints for the nested secrets object.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action: 'Rotate a voice provider integration's secrets.' The verb 'rotate' is specific and distinct from sibling tools like connect, update, get, or delete, making the tool's purpose unambiguous.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description implies usage for rotating secrets but gives no explicit guidance on when to use this tool versus alternatives like update_voice_integration. There is no mention of exclusions or when-not-to-use conditions, though the specific action is self-evident.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
run_simulationRun SimulationAInspect
Run a multi-turn conversation evaluation of your AI agent.
Combines a Target (the system under test), a Driver (the simulated user),
and a Scenario (the test cases) to generate realistic multi-turn conversations
and evaluate them with quality checks.
Returns promptly so the call never times out on long runs. Short runs that
finish within the buffer window return ``status: "finished"`` with results
ready; longer runs return ``status: "running"`` with the ``test_run_id``,
``app_link``, and an ``estimated_runtime`` — the run continues to completion
on its own. In both cases, poll get_test_run_results with the returned
test_run_id for scores, and get_conversation_transcript for transcripts.
To rerun a previous simulation — keeping its configuration but changing one or
more parameters — pass based_on_run_id with the original run's ID and supply
only the values you want to override. If scenario_name or target_name are
omitted and based_on_run_id is provided, they will be resolved from the
original run.
For custom_endpoint Targets: an exception raised during the run (for
example the endpoint erroring mid-conversation) FAILS the run — it is
reported as a failed simulation, not silently skipped.
**Voice augmentations** — for voice Targets, the `augmentation` parameter
applies realistic acoustic and conversational effects. Six top-level keys:
`cap`, `directed_speech`, `secondary_speaker`, `backchannel`, `barge_in`,
plus the composable `noise`. **Composition rule**: at most one non-noise
strategy may be active, optionally combined with `noise`. Augmentations
apply only to voice Targets — calls against generation or custom_endpoint
Targets with an augmentation block are rejected. Field-level errors
(out-of-range probability, missing required field, swapped offsets, unknown
strategy) are returned by the MCP before any backend call.
Strategy required / optional fields (numeric ranges in brackets):
- cap: probability [0.0, 1.0] required. pause_ms [0, 10000] optional.
- directed_speech: probability [0.0, 1.0] required. lpf_cutoff_hz (>0),
gain_db [-40.0, 0.0], sample_rate (>0), prompt, reverb_preset optional.
- secondary_speaker: probability [0.0, 1.0] AND secondary_voice (non-empty
string) required. inter_speaker_pause_ms [0, 5000], lpf_cutoff_hz (>0),
gain_db [-40.0, 0.0], sample_rate (>0), secondary_prompt,
secondary_voice_instructions, secondary_reverb_preset optional.
- backchannel: utterance (non-empty string) required. probability
[0.0, 1.0], min_offset_ms (>=0), max_offset_ms (>= min_offset_ms),
seed optional.
- barge_in: prompt (non-empty string) required. probability [0.0, 1.0],
min_offset_ms (>=0), max_offset_ms (>= min_offset_ms), seed optional.
- noise: noise_profile (non-empty string) AND noise_snr_db (number)
required. seed optional.
For copy-paste examples and the full reference, call
`get_templates(["voice_augmentations"])`.
Args:
name: Human-readable name for this simulation run.
scenario_name: Name of the scenario to use. Required unless based_on_run_id
is provided and the original run's scenario can be resolved.
target_name: Name of the target to evaluate. Required unless based_on_run_id
is provided and the original run's target can be resolved.
driver_name: Name of the driver persona. If omitted, the project default
driver is used.
checks: List of check names to apply (from list_checks). Pick from
the list_checks category matching the task and modality —
voice-specific categories for voice simulations, categories
outside them for either modality; never chat-only checks for
audio (or vice versa). Every simulation runs with at least one
check: when omitted or empty, the benign code-based "latency"
performance check is applied automatically and the response
discloses the substitution via `default_check_applied`.
Supplied checks are used unchanged.
repeats: Number of times to run each scenario row, default 1.
max_turns: Maximum conversation turns per simulation, default 5.
first_turn: Who speaks first — 'target' or 'driver', default 'target'.
based_on_run_id: ID of a previous simulation run to reuse parameters from.
Explicitly supplied values override the original run's parameters.
augmentation: (voice Targets only) Voice augmentation block. See the
"Voice augmentations" section above for keys and ranges. An empty
dict is treated as no augmentation.
turn_transition_time: Milliseconds of pause between turns. Forwarded to
the backend as-is; SDK default (1000) is used when omitted.
silence_timeout_ms: The target reply timeout — how patient Okareo
is before indicating that the target can't respond. Do NOT set
or change this value unless the user specifically directs it;
it should be 10000 ms in nearly all cases. It exists to
accommodate untuned targets with very long tool calls, during
which the Driver waits patiently. It does NOT change how fast
Okareo responds, and lowering it does not speed anything up —
a slow simulation is not a reason to change it. Forwarded to
the backend; backend default is used when omitted.
checks_at_every_turn: When True, checks are evaluated per turn (not
only at end of run).
stop_check: Early-stop config: `{"check_name": str, "stop_on": <value>}`.
The run halts as soon as the named check returns `stop_on`.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | ||
| checks | No | ||
| repeats | No | ||
| max_turns | No | ||
| first_turn | No | target | |
| stop_check | No | ||
| driver_name | No | ||
| target_name | No | ||
| augmentation | No | ||
| scenario_name | No | ||
| based_on_run_id | No | ||
| silence_timeout_ms | No | ||
| checks_at_every_turn | No | ||
| turn_transition_time | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations indicate readOnlyHint=false and openWorldHint=true, and the description adds substantial behavioral context: asynchronous execution ('Returns promptly so the call never times out... run continues to completion on its own'), status values ('finished' vs 'running'), and the failure mode for custom_endpoint targets ('exception raised during the run FAILS the run'). It also discloses default check substitution and field-level validation before backend calls, with no contradiction to annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description front-loads a clear one-sentence summary and is well-structured with sections and an Args list, making it easy to scan. However, it is quite long, and the voice augmentation section provides field-level detail that overlaps with the instruction to call get_templates for the full reference, introducing some redundancy that could be trimmed.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity (14 parameters, async behavior, voice augmentation constraints), the description is remarkably complete. It covers all parameters, return statuses, the polling workflow, failure modes, and check handling. The presence of an output schema means return values need not be repeated, and the description provides all necessary context for successful invocation.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema description coverage is 0%, so the description carries the full burden, and it does thoroughly. The Args section explains every parameter beyond schema types, e.g., 'repeats: Number of times to run each scenario row', 'silence_timeout_ms: Do NOT set... should be 10000 ms', and the augmentation block is fully detailed with required/optional fields and numeric ranges. This far exceeds minimal compensation for the schema gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'Run a multi-turn conversation evaluation of your AI agent,' a specific verb+resource+modality that clearly states the tool's purpose. It distinguishes from siblings by emphasizing multi-turn conversation evaluation and pointing to follow-up tools like get_test_run_results and get_conversation_transcript, setting it apart from run_test and other simulation-related tools.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides explicit when-to-use and when-not-to-use guidance: silence_timeout_ms should not be changed unless requested, augmentation is rejected for non-voice targets, and checks are sourced from list_checks. It also directs users to get_templates for examples and get_test_run_results for polling, clearly naming alternatives and context for use.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
run_testRun TestAInspect
Submit a quality test that evaluates a model against a scenario using checks.
Returns promptly so the call never times out on long runs. Short runs return
``status: "finished"`` with results ready; longer runs return
``status: "running"`` with the ``test_run_id`` and ``app_link`` — the run
continues to completion on its own. In both cases, poll get_test_run_results
with the returned test_run_id to retrieve scores.
Args:
scenario_name: Name of the scenario to evaluate against.
model_name: Name of the registered model to evaluate.
checks: List of check names to apply (e.g., ["coherence", "relevance"]).
Use list_checks to discover available checks and pick from the
category matching the task and modality — do not use
voice-specific checks for text evaluations (or vice versa);
checks outside voice-specific categories suit both.
name: Optional human-readable name for this test run.
type: Type of evaluation. Defaults to NL_GENERATION. Valid values:
NL_GENERATION, INFORMATION_RETRIEVAL, MULTI_CLASS_CLASSIFICATION,
INVARIANT, MULTI_TURN, AGENT_EVAL.
| Name | Required | Description | Default |
|---|---|---|---|
| name | No | ||
| type | No | NL_GENERATION | |
| checks | Yes | ||
| model_name | Yes | ||
| scenario_name | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already mark this as non-read-only and non-idempotent, but the description adds essential async behavior: it returns promptly, may return status 'finished' or 'running' with test_run_id and app_link, and continues on its own. This fully discloses the lifecycle and interaction pattern.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is compact but dense, with no fluff. The Args block is a simple list within a paragraph, which could be more readable as a bulleted list, but every sentence earns its place and it avoids redundancy with the schema.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a 5-parameter tool with an output schema present, the description covers all dimensions: what it does, how to use it, async behavior, parameter semantics, and interaction with sibling tools. The output schema handles return value details, so no additional return documentation is needed.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
Schema parameter descriptions are absent (0% coverage), so the description carries full responsibility. It defines every parameter: scenario_name, model_name, checks with an example and guidance on selection, name as optional, and type with all valid enum values. This compensates completely for the schema gap.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a specific verb+resource: 'Submit a quality test that evaluates a model against a scenario using checks.' This clearly distinguishes run_test from siblings like run_simulation or get_test_run_results.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explains when to use this tool for submitting a test, notes that short runs finish immediately while long runs return a status and ID, and explicitly directs polling via get_test_run_results. It also advises using list_checks and warns against mismatched check categories, providing actionable when-to-use guidance.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
save_dashboardSave DashboardAIdempotentInspect
Create or update an analytics dashboard by name (upsert).
If a dashboard with this name already exists it is updated; otherwise a
new one is created.
Size each panel with a named ``size`` from the catalog below (PREFERRED
— guarantees a legible layout) and omit positions entirely: panels are
auto-placed in the order given (left-to-right, top-to-bottom on a
12-column grid, wrapping rows, never overlapping).
Size catalog and when to use each:
- ``small-square`` (3x6): single ``stat`` metrics.
- ``half-rectangle`` (6x6): ``line``/``bar``/``area`` trends, two per row.
- ``half-square`` (6x9): ``radar``, ``composed``, denser charts.
- ``full-rectangle`` (12x9): wide time-series comparisons.
- ``full-square`` (12x12): ``table`` panels.
Args:
name: Dashboard name — the upsert key.
panels: Optional list of panel definitions. Each panel is an object:
- ``title`` (str, required): panel heading.
- ``chart_type`` (str, required): one of ``line``, ``bar``,
``composed``, ``area``, ``radar``, ``stat``, ``table``.
- ``query`` (object, required): what to chart —
``{"cube": "check_trend", "measures": [...],
"dimensions": [...], "filters": [...],
"time_dimensions": [...], "order": {...}}``. ``measures`` is
required; everything else is optional. ``cube`` defaults to
``check_trend``. The dashboard ``time_range`` applies to all
panels — panels do NOT carry their own time range.
- ``size`` (str): a catalog name (see above). Required unless
``layout`` is given; wins over ``layout`` w/h if both appear.
- ``layout`` (object): raw grid placement
``{"x": >=0, "y": >=0, "w": >=1, "h": >=1}`` (integers).
Only needed when not using ``size``, or to pin an explicit
position (give both ``x`` and ``y``; with ``size``, w/h are
ignored). Heights below the legibility floor are sized up on
save: ``h >= 6`` when ``w <= 6``, ``h >= 9`` when ``w > 6``.
- ``table_config`` (object, optional): ONLY for
``chart_type == "table"``.
``check_trend`` measures: ``avg_check_value``, ``issue_rate``,
``error_rate``, ``datapoint_count``, ``issue_count``,
``error_count``, ``test_run_count``, ``avg_latency``,
``sum_cost``, ``input_token_count``, ``output_token_count``.
``check_trend`` dimensions: ``check.name``, ``check.id``,
``target.name``, ``target.id``, ``scenario.name``,
``scenario.id``, ``test_run.id``, ``test_run.type``,
``test_run.is_latest_for_target``, ``source``, ``provider``,
``request_model_name``, ``response_model_name``, ``tag``.
Use ``query_analytics(include_metadata=True)`` for the
authoritative, current set.
Example panel::
{"title": "Avg Check Value by Check", "chart_type": "bar",
"query": {"measures": ["avg_check_value"],
"dimensions": ["check.name"]},
"size": "half-rectangle"}
description: Optional dashboard description.
time_range: Optional default look-back window for the whole
dashboard. One of: LAST_HOUR, LAST_24_HOURS, LAST_7_DAYS,
LAST_14_DAYS, LAST_30_DAYS, LAST_90_DAYS. Defaults to
LAST_90_DAYS when omitted.
Returns:
JSON with the saved dashboard and ``action`` (created/updated).
When sizing or dimensions were changed on save (size overriding
layout w/h, or a height floored), an ``adjustments`` list reports
each change: ``{"panel", "field", "from", "to", "reason"}``.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | ||
| panels | No | ||
| time_range | No | ||
| description | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Beyond the annotations (idempotentHint=true, destructiveHint=false), the description details many behavioral traits: upsert behavior, auto-placement of panels, size overriding layout w/h, height flooring adjustments, and the return of an 'adjustments' list. This adds substantial context beyond what annotations provide.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is long but well-structured with clear sections, bullet points, and an example. Every sentence adds value, though it could be slightly tightened. The front-loaded first sentence states the core purpose immediately.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given the tool's complexity and the sparse schema, the description is exhaustive: it explains return values (including 'action' and 'adjustments'), defaults (time_range defaults to LAST_90_DAYS), the query object, and how to get authoritative dimensions/measures. It fully compensates for the lack of schema descriptions.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema is extremely sparse (0% coverage), so the description carries the full burden. It thoroughly explains each parameter: name, panels (with full panel object structure including title, chart_type, query, size, layout, table_config), time_range allowed values, and description. It also enumerates measures/dimensions and provides an example.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'Create or update an analytics dashboard by name (upsert)', which uses a specific verb ('create or update'), identifies the resource ('analytics dashboard'), and highlights the upsert behavior. This clearly distinguishes it from siblings like get_dashboard, list_dashboards, and delete_dashboard.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides clear guidance on when to use the tool (upsert semantics) and internal usage guidelines such as the size catalog and when to use size vs. layout. However, it does not explicitly contrast with sibling tools (e.g., 'use get_dashboard to view rather than save'), though the purpose statement implies it.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
save_scenarioSave ScenarioAIdempotentInspect
Save a named scenario for use in quality tests.
Provide EXACTLY ONE dataset source: `content`, `file_path`, or
`rows`.
Prefer `file_path` for local .jsonl files — the server reads the
file directly, so no rows pass through the assistant's context.
When passing rows through the assistant instead, keep the dataset
UNDER 2,000 rows (`content` with the file's text, or `rows` for
tiny datasets). For 2,000 rows or more, always use `file_path` or
upload directly to Okareo via the web app, SDK, or CLI, to avoid
unnecessary token cost.
If a scenario with the same name already exists, the existing
scenario is returned (idempotent). Scenarios are immutable after
creation — use create_scenario_version to create updated versions.
Args:
name: A unique name for the scenario.
content: Raw JSONL text (one JSON object with 'input' and
'result' per line). Only for datasets under 2,000 rows.
file_path: Path to a local .jsonl file. Preferred — works for
any size.
rows: List of data rows, each with 'input' (any type) and
'result' (any type). Use for small scenarios (< 20 rows).
tags: Optional list of tags for categorizing the scenario.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | ||
| rows | No | ||
| tags | No | ||
| content | No | ||
| file_path | No |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Confirms idempotent behavior (existing scenario returned) and adds that scenarios are immutable after creation. Also discloses that file_path avoids passing rows through assistant context and notes token cost considerations, going beyond the idempotentHint annotation.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The structure is well-organized: opening purpose, usage guidance, idempotency note, and a clear Args list. Each sentence provides necessary information without unnecessary fluff, though slightly long due to parameter guidance.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Covers parameter selection, constraints, idempotency, immutability, and alternatives. Given that an output schema exists, it needn't describe return values; it's complete for a tool of this complexity.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
With 0% schema coverage, the description provides detailed semantics for all 5 parameters, including constraints (e.g., 'one JSON object with input and result per line' for content, '< 20 rows' for rows) and trade-offs between sources.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states 'Save a named scenario for use in quality tests,' using a specific verb and resource. It distinguishes itself from sibling tools by explicitly noting that scenarios are immutable and that create_scenario_version is used for updates.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
Provides explicit guidance on providing EXACTLY ONE dataset source, with clear preferences (file_path for local files and large datasets, content/rows for small datasets under 2,000 rows). It also references alternative upload methods (web app, SDK, CLI) and points to create_scenario_version for updates.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
switch_tenantSwitch TenantBRead-onlyIdempotentInspect
Change which Okareo organization your session operates against.
Organization selection now happens **during sign-in** (feature 030):
when you connect the Okareo MCP you choose which organization to
authorize, and the credential this session uses is already scoped to
it. This tool therefore no longer changes the active organization — to
switch, reconnect/re-authenticate the Okareo MCP from your copilot and
select a different organization when prompted. Use ``list_tenants`` to
see which organization is currently active.
| Name | Required | Description | Default |
|---|---|---|---|
| tenant_id | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
The description clearly discloses that the tool no longer changes the active organization and explains why (feature 030, credential scoped at sign-in). This goes beyond the annotations (readOnlyHint, destructiveHint, idempotentHint) by providing historical context and the reason for the no-op behavior. It stops short of describing what the tool returns or whether it throws an error, but the core behavioral trait is well covered.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is a single, focused paragraph that explains the deprecation and provides alternatives. While it includes some background detail (feature number), every sentence serves to prevent misuse. It is appropriately sized for a deprecated tool, though it could be slightly tightened by removing the parenthetical 'feature 030'.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
Given that the tool is essentially deprecated, the description covers the necessary context: it explains the change, tells the agent how to actually switch, and points to list_tenants. An output schema exists (unseen) which may describe return values, so the lack of return format details in the description is acceptable. It would be more complete if it explicitly stated 'this tool now does nothing' or described the effect of calling it.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema has 0% description coverage for the single parameter tenant_id, and the description never mentions it. The agent gets no insight into what this parameter is for, how it should be formatted, or whether it is still relevant given the tool's deprecated behavior. The description fails to compensate for the lack of schema documentation.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with 'Change which Okareo organization your session operates against' but immediately contradicts itself by stating 'This tool therefore no longer changes the active organization.' It never clearly states what the tool actually does when invoked now, leaving the agent uncertain whether it is a no-op, a read-only status check, or an error. This is not a tautology but is misleading about the tool's current function.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description explicitly tells the agent to reconnect/re-authenticate the Okareo MCP to switch organizations and to use list_tenants to see the active organization. This provides clear exclusions and alternatives. However, it offers no guidance on when to actually call switch_tenant, making it more of a deprecation notice than a usage guide.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
update_generation_modelUpdate Generation ModelAIdempotentInspect
Change the LLM that a registered generation model points to.
Updates the generation model to use a different LLM from the registry.
Use list_available_llms to see available LLMs.
Args:
name: Name of the registered generation model to update.
model_name: The new LLM from the registry.
| Name | Required | Description | Default |
|---|---|---|---|
| name | Yes | ||
| model_name | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already indicate readOnlyHint=false, idempotentHint=true, and destructiveHint=false. The description adds useful context by clarifying that the update changes the model to point to a different LLM 'from the registry' and implies that the new model must come from list_available_llms. This is sufficient for a simple update operation, though it does not discuss error cases or validation behavior.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and well-structured: an opening summary sentence, a practical pointer to a related tool, and a clean Args section. Every sentence serves a purpose, and there is no redundant or filler content.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
For a simple two-parameter update tool that has an output schema and good annotations, the description is complete. It states what the tool does, the source of valid values, and explains both parameters. No further behavioral or return-value detail is needed given the low complexity and available structured metadata.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The input schema provides no descriptions (0% coverage), so the description must compensate. The Args section clearly explains both parameters: 'name' is the registered generation model to update, and 'model_name' is the new LLM from the registry. This adds real meaning beyond the bare property names, even if it does not specify value formats.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description opens with a clear, specific action: 'Change the LLM that a registered generation model points to.' This precisely identifies the verb (change/update), the resource (generation model), and the specific attribute being modified (LLM). It clearly distinguishes this from sibling tools like register_generation_model, get_generation_model, and delete_generation_model.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
The description provides a clear usage hint: 'Use list_available_llms to see available LLMs.' This guides the agent to a prerequisite/related tool. It does not explicitly state when not to use this tool or list alternative update scenarios, so it falls just short of a 5.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
update_voice_integrationUpdate Voice IntegrationAIdempotentInspect
Update a voice provider integration's metadata.
Args:
integration_id: The integration's id.
metadata: The new metadata object.
| Name | Required | Description | Default |
|---|---|---|---|
| metadata | Yes | ||
| integration_id | Yes |
Output Schema
| Name | Required | Description |
|---|---|---|
| result | Yes |
TDQS
Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?
Annotations already declare idempotentHint=true, readOnlyHint=false, and destructiveHint=false, but the description adds no extra behavioral context beyond restating the update operation. It does not explain whether the metadata is fully replaced or merged, or any side effects, providing minimal value over the annotations.
Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.
Is the description appropriately sized, front-loaded, and free of redundancy?
The description is concise and to the point, with a clear opening sentence followed by a minimal Args section. There is no redundancy or wasted wording, making it easy to scan and understand.
Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.
Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?
While the core purpose and parameters are covered, the description omits important contextual details like merge-vs-replace behavior, potential constraints on metadata, or prerequisites. Given the output schema exists, return values are not needed, but the description is otherwise minimally adequate for a simple update tool.
Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.
Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?
The schema coverage is 0%, so the description is responsible for explaining parameters. It provides basic definitions for integration_id and metadata, but lacks deeper semantics such as the expected structure of the metadata object or whether it is a partial update, leaving significant room for clarification.
Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.
Does the description clearly state what the tool does and how it differs from similar tools?
The description clearly states the action ('Update') and the target resource ('a voice provider integration's metadata'), using a specific verb+resource structure. It distinguishes itself from sibling voice integration tools like connect, rotate, and delete by focusing on metadata updates.
Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.
Does the description explain when to use this tool, when not to, or what alternatives exist?
No explicit guidance is provided on when to use this tool versus alternatives. The usage is only implied by the tool name and description, offering no exclusions or alternative recommendations.
Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
51 tool updates
v0.0.43- First observed
connect_voice_integration - First observed
create_or_update_check - First observed
create_or_update_driver - First observed
create_or_update_target - First observed
create_scenario_version - First observed
delete_check - First observed
delete_dashboard - First observed
delete_generation_model - First observed
delete_scenario - First observed
delete_target - First observed
delete_voice_integration - First observed
generate_check - First observed
get_check - First observed
get_conversation_transcript - First observed
get_dashboard - First observed
get_docs - First observed
get_driver - First observed
get_generation_model - First observed
get_reps_baseline - First observed
get_scenario - First observed
get_target - First observed
get_templates - First observed
get_test_run_results - First observed
get_voice_integration - First observed
get_voice_webhook_url - First observed
ingest_conversations - First observed
list_available_llms - First observed
list_checks - First observed
list_dashboards - First observed
list_driver_voices - First observed
list_drivers - First observed
list_generation_models - First observed
list_scenarios - First observed
list_simulations - First observed
list_targets - First observed
list_tenants - First observed
list_test_runs - First observed
list_voice_integrations - First observed
preview_delete_scenario - First observed
query_analytics - First observed
reevaluate_test_run - First observed
register_generation_model - First observed
reorder_dashboards - First observed
rotate_voice_integration_secret - First observed
run_simulation - First observed
run_test - First observed
save_dashboard - First observed
save_scenario - First observed
switch_tenant - First observed
update_generation_model - First observed
update_voice_integration
TDQS
Scored across 51 tools
Most tools are clearly distinct by resource and action, with only a few potentially confusing pairs like run_test vs run_simulation or register_generation_model vs create_or_update_target with a 'generation' type. Detailed descriptions and distinct parameters largely resolve ambiguity, so an informed agent can reliably select the right tool.
The set predominantly follows a verb_noun pattern (list_, get_, create_or_update_, delete_) with consistent snake_case. Minor deviations exist, such as save_scenario/save_dashboard instead of create_, register_generation_model instead of create_generation_model, and longer phrases like get_voice_webhook_url, which keep it from being perfectly uniform.
At 51 tools, the server is well beyond the 25+ threshold considered too many for a single MCP server. While the broad scope of the Okareo platform explains the abundance, the sheer size will overwhelm agents and make tool selection and discovery unnecessarily difficult.
The server covers a wide range of resources with robust CRUD support, but there are notable gaps: there is no delete_driver tool, and no way to delete test runs. The driver lifecycle in particular is incomplete, which is a significant omission for an otherwise comprehensive platform surface.
Maintenance
Related MCP Connectors
The OpenRouter MCP server plugs OpenRouter into the AI tools you already use. Once connected, your assistant can pull live OpenRouter data (models, prices, your credits, rankings, and docs) and send quick test messages, all without leaving your editor.
Your org's AI agents, tasks, runs, search, and brain files as MCP tools and resources.
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
MCP server that lets AI assistants use all OneSchema features exposed via the public API.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceExposes Language Server Protocol (LSP) functionality as Model Context Protocol (MCP) tools, enabling AI clients to programmatically analyze and edit code in any language supported by VS Code.633MIT

Patronus MCP Serverofficial
AlicenseNot gradedqualityDmaintenanceEnables running LLM evaluations, experiments, and custom evaluators through a standardized MCP interface.16Apache 2.0- FlicenseNot gradedqualityDmaintenanceEnables LLM-powered code analysis, generation, debugging, and context management through MCP integration with IDEs like Cursor and Claude Desktop.-
- AlicenseNot gradedqualityAmaintenanceEnables LLM evaluation and observability by uploading documents, building test sets, running RAG pipelines, and automatically scoring answers for groundedness, hallucination risk, retrieval quality, latency, and cost, with tools exposed to MCP-compatible clients.1MIT