Peek-safe A/B verdict
ab_testCompare two cumulative binary-outcome experiment arms to decide whether either success rate is higher. Use for conversions or other Bernoulli outcomes with constant conditional success probabilities; use did_it_change for an ordered numeric metric, not these counts. The beta-binomial confidence sequences split alpha across arms, allowing repeated looks under the stated sampling assumptions. Do not substitute selected windows, continuous returns or changing-probability outcomes. Returns decided, winner (label or null), reading, and each label's successes/n/rate/interval; rates and bounds are fractions in [0,1]. Zero totals are allowed and leave the test undecided. No winner means overlapping intervals, not equal rates. Invalid counts (including successes > total) or duplicate/reserved labels return a tool error. Example: {"a_successes":40,"a_total":100,"b_successes":55,"b_total":100}.
Tool call template
Replace each placeholder with a value matching the input schema. Only required parameters are included; add optional parameters as needed.
{
"id": 1,
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"arguments": {
"a_successes": "<a_successes>",
"a_total": "<a_total>",
"b_successes": "<b_successes>",
"b_total": "<b_total>"
},
"name": "ab_test"
}
}Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| alpha | No | Total error budget shared by both confidence sequences; default 0.05 means 5%, split equally between arms. Must be strictly between 0 and 1. | |
| a_label | No | Arm A's output key and possible winner label; default A. Must differ from b_label and not replace result keys such as decided, winner or reading. | A |
| a_total | Yes | Cumulative trials in arm A, including successes and failures. Zero is valid and returns an undecided result. | |
| b_label | No | Arm B's output key and possible winner label; default B. Must differ from a_label and not replace result keys such as alpha, method or validity. | B |
| b_total | Yes | Cumulative trials in arm B, including successes and failures. Counts must use the same pre-defined experiment and outcome as arm A. | |
| a_successes | Yes | Cumulative successful binary outcomes in arm A, an integer from 0 through a_total; do not supply a percentage. | |
| b_successes | Yes | Cumulative successful binary outcomes in arm B, an integer from 0 through b_total; do not supply a percentage. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| alpha | No | Total error budget split across the arms, as a fraction. | |
| error | No | A string on tool error, or a successful arm object when the caller labels that arm error; inspect MCP isError. | |
| method | No | Confidence-sequence method and reference. | |
| winner | No | Winning caller-supplied arm label; null when undecided. Null means unavailable or not applicable, never zero. | |
| decided | No | Whether the two confidence sequences are disjoint. | |
| reading | No | Interpretation of the current comparison. | |
| validity | No | Guarantees are conditional on the Bernoulli sampling model. | |
| guarantee | No | Conditional confidence-sequence guarantee; not a market-performance claim. | |
| assumptions | No | Ordered statements or identifiers. | |
| safe_to_peek | No | Repeated looks are allowed only under the stated Bernoulli assumptions. | |
| what_would_decide_it | No | Qualitative guidance for an undecided comparison with data in both arms. | |
| observed_difference_pp | No | B's observed success rate minus A's, in percentage points; present only for undecided nonempty arms. |