ab_test_report
Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percentage, and a Welch t-test at 95 %: winner names a variant only when the difference is significant — otherwise "no_significant_difference", or "insufficient_samples" under 2 scores per variant.
Example tool call
This call uses example arguments supplied in the tool’s input schema.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"variant_a": {
"name": "A",
"scores": [
70,
71,
69
]
},
"variant_b": {
"name": "B",
"scores": [
70,
71,
69
]
}
},
"name": "ab_test_report"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| variant_a | Yes | First variant configuration with name and score array | |
| variant_b | Yes | Second variant configuration with name and score array |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| max | No | ||
| min | No | ||
| mean | No | ||
| count | No | ||
| median | No | ||
| winner | No | ||
| std_dev | No | ||
| variant_a | No | ||
| variant_b | No | ||
| recommendation | No | ||
| improvement_percent | No |