Welch's t-Test, Two-Proportion z-Test & Bayesian A/B Eval Judge
evaluate_ab_statistical_significanceComputes Welch's two-sample t-test or two-proportion z-test, returning two-tailed p-values, Cohen's d effect size, relative lift %, and Bayesian P(Treatment > Control). Use when determining whether a prompt change, model benchmark, or A/B experiment produced a statistically significant improvement; for evaluating single algebraic formulas or NPV/IRR cashflows, use evaluate_deterministic_math instead. Read-only edge call (32 KB, 60 RPM): accepts 'control_json' ('treatment_json' defaults to 'standard'), supports '?demo=true' or L402 (60 sats), and returns HTTP 400 if invalid.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| control_json | Yes | JSON string or object for the baseline control arm (e.g. '{"conversions":142,"total":1000}' or '{"samples":[12,15,14]}'). | |
| treatment_json | Yes | JSON string or object for the candidate treatment arm (e.g. '{"conversions":186,"total":1000}'). | |
| significance_level | No | Type-I error alpha threshold for statistical significance; defaults to 'alpha_0_05' when omitted. |
Output Schema
| Name | Required | Description | Default |
|---|---|---|---|
| p_value | Yes | Two-tailed frequentist p-value. | |
| verdict | Yes | Experiment decision ('SIGNIFICANT_TREATMENT_WIN', 'SIGNIFICANT_CONTROL_WIN', or 'INCONCLUSIVE_UNDERPOWERED'). | |
| test_type | Yes | Statistical test applied ('welch_two_sample_t_test' or 'two_proportion_z_test_and_bayesian'). | |
| relative_lift_pct | Yes | Percentage lift of treatment over control. | |
| bayesian_prob_treatment_superior | Yes | Posterior probability P(Treatment > Control) from 0.0 to 1.0. |