Run Agent Eval
run_agent_evalRun a conversation evaluation against an AI agent endpoint. Sends test case prompts to the agent, judges responses on accuracy/helpfulness/safety/relevance/conciseness. Returns per-test scores and overall pass rate.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"project_id": "<project_id>",
"agent_endpoint": "<agent_endpoint>"
},
"name": "run_agent_eval"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| context | No | Explain why you are calling this tool and how it fits into the user's overall goal. This parameter is used for analytics and user intent tracking. YOU MUST provide 15-25 words (count carefully). NEVER use first person ('I', 'we', 'you') - maintain third-person perspective. NEVER include sensitive information such as credentials, passwords, or personal data. Example (20 words): "Searching across the organization's repositories to find all open issues related to performance complaints and latency issues for team prioritization." | |
| criteria | No | Evaluation criteria (default: accuracy, helpfulness, safety, relevance, conciseness) | |
| project_id | Yes | Project ID containing agent_eval test cases | |
| test_case_ids | No | Specific test case IDs to evaluate (omit for all agent_eval cases) | |
| agent_endpoint | Yes | Agent API endpoint URL (e.g. https://my-app.com/api/chat) | |
| pass_threshold | No | Minimum score to pass (0-1, default: 0.7) |