Run Agent Eval
run_evalRun saved test scenarios against an agent's live configuration and grade replies by eval_id or agent_id, returning pass/fail with reasons without calls or config changes.
Instructions
Run a saved test scenario against one of the business's agents and grade the agent's replies: pass eval_id for one scenario, or agent_id for every scenario saved for that agent (see list_evals). The test is a simulated text conversation with the agent's current live configuration; no phone call is placed, no customer is contacted and the agent's configuration is not changed. Each run uses paid voice-agent provider capacity and is recorded with its result, so scenarios already run on the agent's current configuration are not run again unless force is true, and each agent has a daily limit on runs. Returns pass or fail per scenario with the grader's reasons.
Tool call template
This tool declares no required parameters. Add optional parameters from the input schema as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {},
"name": "run_eval"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| wait | No | Wait for the grade before returning (default true). | |
| force | No | Run again even though nothing changed on the agent or these scenarios since they last ran (otherwise refused, since the result would be the same). | |
| eval_id | No | Run this one saved scenario (from list_evals). Either this or agent_id is required. | |
| agent_id | No | Run every saved scenario for this agent. Either this or eval_id is required. | |
| max_wait_ms | No | Per-run wait cap in ms when wait is true (default 60000, max 110000). |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| error | No | Present when success is false | |
| total | No | ||
| failed | No | ||
| passed | No | ||
| run_id | No | ||
| status | No | ||
| results | No | ||
| success | Yes | Whether the tool completed successfully | |
| completed | No | ||
| provider_run_id | No |