backtest_output
Validate a simulation model's predicted distribution against realized actuals by computing Probability Integral Transform, interval coverage, and bias.
Instructions
ModelRisk: Backtest a simulation output against realised actuals — does the model's predicted distribution match what actually happened? Reports the Probability Integral Transform (PIT, ~0.5 mean and uniform if calibrated), the empirical coverage of central prediction intervals (e.g. ~90% of actuals should fall in the 90% interval), and bias. Pass the historical actuals you want to validate against. Reads the output's per-iteration samples — run the simulation first.
Input Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| max_n | No | Max samples to read (default 100 000). | |
| actuals | Yes | Realised historical values. | |
| intervals | No | Central intervals to check coverage of. Default [0.5,0.8,0.9,0.95]. | |
| output_name | Yes | VoseOutput name to validate. | |
| workbook_name | No | Workbook name. Omit for the active workbook. |
Output Schema
TableJSON Schema
| Name | Required | Description | Default |
|---|---|---|---|
| bias | Yes | actuals_mean - model_mean. | |
| verdict | Yes | Short calibration verdict. | |
| coverage | Yes | Empirical vs nominal coverage of central prediction intervals. | |
| mean_pit | Yes | Mean Probability Integral Transform; ~0.5 if calibrated. | |
| n_actuals | Yes | ||
| model_mean | Yes | ||
| output_name | Yes | ||
| sample_size | Yes | ||
| actuals_mean | Yes | ||
| frac_below_median | Yes | Fraction of actuals below the model median; ~0.5 if calibrated. | |
| pit_uniformity_ks | Yes | KS distance of the PIT values from Uniform(0,1); 0 = perfectly calibrated. |