eval_answer_accuracy
Evaluates semantic equivalence between an answer and ground truth using QAG grading. Returns score, pass/fail, and reason.
Instructions
Evaluate whether an answer is semantically equivalent to the ground truth.
QAG-graded — generates yes/no questions about whether the actual answer matches the meaning of the expected answer. Useful when string match is too strict (e.g. paraphrased correct answers).
Args: expected_answer: Ground-truth answer. actual_answer: The LLM's answer. judge_model: Provider:model for the QAG judge.
Returns:
{"score": 0.0-1.0, "passed": bool, "reason": str}.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"expected_answer": "<expected_answer>",
"actual_answer": "<actual_answer>"
},
"name": "eval_answer_accuracy"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| judge_model | No | anthropic:claude-haiku-4-5 | |
| actual_answer | Yes | ||
| expected_answer | Yes |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||