adversarial_check
Test a finished backtest run against six controls plus a positive control to confirm whether its edge survives or is refuted by the data.
Instructions
Requires the Stochastly desktop app: this tool runs inside the app, which is not installed on this machine. Install it from https://stochastly.com/pricing, open it and activate a licence.
Run six controls that try to refute a finished run (drift against its own block-bootstrap null, an edge-free twin of the prop-firm cycle, Romano-Wolf multiplicity, concentration on the best days, frictions, stability across sub-periods) plus a positive control that checks a known edge would be detected on the same data.
Call it after any promising result. synthesis is SURVIVES, REFUTED, UNDERPOWERED (effect
smaller than the sample can resolve; observations_needed says how many bars it would take)
or BLIND (the positive control failed, so the sample, not the strategy, must change). Each
passed control reports its floor, the smallest effect it could have rejected. Deterministic
arithmetic, no language model; measured error rates in the test suite: 0.017 false
positives, 0.000 false negatives. Needs the run's full PnL series with at least 8 points; an
unknown run is a named error. Needs Stochastly open with an Atelier or Portfolio licence.
Input Schema
| Name | Required | Description | Default |
|---|---|---|---|
| seed | No | Random seed; the same seed gives the same answer. | |
| n_boot | No | Number of bootstrap resamples. | |
| n_sims | No | Number of simulations for the randomised controls. | |
| run_id | Yes | Id of a finished run, from backtest_graph or backtest_status. |