eval_compare_paired
Compare two models on the same test set by running an exact McNemar test on paired outcomes and returning a p-value plus a confidence interval for the accuracy gap.
Instructions
Is A's win over B on a shared test set statistically real?
n: total cases; a_only: A right & B wrong; b_only: B right & A wrong;
both: both right. Returns exact McNemar p-value and a CI on the gap.Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"n": "<n>",
"a_only": "<a_only>",
"b_only": "<b_only>",
"both": "<both>"
},
"name": "eval_compare_paired"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| n | Yes | ||
| both | Yes | ||
| alpha | No | ||
| a_only | Yes | ||
| b_only | Yes |