Skip to main content
Glama
KaiCalls MCP Server

KaiCalls MCP Server

Official

Run Agent Eval

run_eval
Destructive

Run saved test scenarios against an agent's live configuration and grade replies by eval_id or agent_id, returning pass/fail with reasons without calls or config changes.

Instructions

Run a saved test scenario against one of the business's agents and grade the agent's replies: pass eval_id for one scenario, or agent_id for every scenario saved for that agent (see list_evals). The test is a simulated text conversation with the agent's current live configuration; no phone call is placed, no customer is contacted and the agent's configuration is not changed. Each run uses paid voice-agent provider capacity and is recorded with its result, so scenarios already run on the agent's current configuration are not run again unless force is true, and each agent has a daily limit on runs. Returns pass or fail per scenario with the grader's reasons.

Tool call template

This tool declares no required parameters. Add optional parameters from the input schema as needed.

{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "arguments": {},
    "name": "run_eval"
  }
}

Input Schema

NameRequiredDescriptionDefault
waitNoWait for the grade before returning (default true).
forceNoRun again even though nothing changed on the agent or these scenarios since they last ran (otherwise refused, since the result would be the same).
eval_idNoRun this one saved scenario (from list_evals). Either this or agent_id is required.
agent_idNoRun every saved scenario for this agent. Either this or eval_id is required.
max_wait_msNoPer-run wait cap in ms when wait is true (default 60000, max 110000).

Output Schema

NameRequiredDescriptionDefault
errorNoPresent when success is false
totalNo
failedNo
passedNo
run_idNo
statusNo
resultsNo
successYesWhether the tool completed successfully
completedNo
provider_run_idNo

Schema Changelog

Changes observed during successful MCP inspections.

  1. First observedv0.1.0

TDQS

A4.6/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes well beyond the annotations: it rules out a real phone call, customer contact, and configuration changes, and discloses cost (paid provider capacity), bounding (per-agent daily run limit), and result persistence/dedup semantics. This resolves the tension an agent might feel between destructiveHint=true and the reassurance that nothing live is modified.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loads what the tool does and the id-selection rule before moving to side effects and cost. Dense and largely waste-free, though the force/dedup rationale is stated twice (once in prose, once in the schema) and could be trimmed.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, yet the description still summarizes the shape of results (pass/fail per scenario with grader reasons). Combined with cost, limits, and no-side-effect disclosure, an agent has everything required to call this correctly and warn about billing implications.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so eval_id, agent_id, force, wait, and max_wait_ms are already documented, including default and max values. The prose restates the either/or requirement and the force rationale rather than adding new meaning, so it stays at the high-coverage baseline.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Run a saved test scenario against one of the business's agents and grade the agent's replies') and immediately clarifies the two targeting modes (eval_id for one, agent_id for all). It also names the sibling used to obtain ids (list_evals), so an agent can distinguish it from catalog-management tools without opening a schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly says when to use which parameter, points to list_evals for discovery, and states a clear when-not condition: scenarios already run against the current configuration are refused unless force is true. The preconditions (paid capacity, daily limit) give an agent everything needed to decide whether to call it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Deploy Server

Other Tools