Compare two models' per-example scores
eval_compareCompare two models on the same paired scores to get mean difference, bootstrap confidence intervals, p-values, win/tie counts, and McNemar tests for binary outcomes.
Instructions
Paired comparison of model A and model B on the same examples: mean difference (B minus A), seeded paired bootstrap confidence interval and two-sided p-value, win/tie counts, and McNemar's test (exact and chi-square) when every score is 0 or 1.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"scoresA": "<scoresA>",
"scoresB": "<scoresB>"
},
"name": "eval_compare"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| seed | No | Random seed (default 0) | |
| scoresA | Yes | Per-example scores for model A | |
| scoresB | Yes | Per-example scores for model B, same order | |
| confidenceLevel | No | Confidence level (default 0.95) | |
| bootstrapSamples | No | Bootstrap resamples (default 10000) |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| n | Yes | ||
| ties | Yes | ||
| meanA | Yes | ||
| meanB | Yes | ||
| winsA | Yes | ||
| winsB | Yes | ||
| mcnemar | Yes | ||
| bootstrap | Yes | ||
| standardError | Yes | ||
| interpretation | Yes | ||
| meanDifference | Yes |