eval_relevance
Evaluates whether an LLM response answers the user's question by generating and scoring yes/no questions.
Instructions
Check whether an LLM output actually addresses the user's question.
QAG-graded — generates yes/no questions about whether the output answers the input, stays on topic, contains relevant content.
Args: input: The user's question. output: The LLM's response. judge_model: Provider:model for the QAG judge.
Returns:
{"score": 0.0-1.0, "passed": bool, "reason": str, "threshold": float}.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"input": "<input>",
"output": "<output>"
},
"name": "eval_relevance"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| input | Yes | ||
| output | Yes | ||
| judge_model | No | anthropic:claude-haiku-4-5 |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
No arguments | |||