# backtest-reality-check

**An MCP server that tells you whether your backtest survives its own statistics.**

Point any MCP client at it and ask "is my backtest real?". It answers with
arithmetic — deflated Sharpe, minimum track record length, multiple-testing
correction, fill realism — using the checks professional quant research applies
and hobby trading bots almost never do.

No market data. No API keys. No broker connection. It cannot recommend a trade
and has nothing to sell. It can only ever tell you your number is weaker than you
think, or that you can't tell yet. Those are the only two honest answers available.

**▶ [60-second demo](https://github.com/maximdakulinin-boop/backtest-reality-check/raw/main/demo.mp4)** — the simulation, the maths, and the tool failing on my own dead bot.

---

## Why it exists

I built a sports-betting engine. The backtest reported a **Sharpe ratio of 8.61**.
It went **0 for 38** live.

Two things were wrong, and only one of them is the famous one:

1. **Overfitting.** I'd tried ~25 variants and kept the best. Nobody counts that.
2. **Broken data.** The P&L was mis-computed and the fills were at prices the
   market never offered — 25.5% of them better than anything actually available.

This tool catches both, with two different detectors, because **they are different
failures and a Sharpe ratio can only see one of them.**

## The honest limitation, stated up front

**Deflation catches overfitting and luck. It cannot detect broken data.**

Run my own dead bot through `check_backtest` — Sharpe 8.61, 0.3 years, 25 trials —
and it returns **SURVIVES at 99.4%**. Because a Sharpe that high genuinely *is*
extraordinary *if the returns are real*. Mine weren't.

That is why `check_fills` exists. **A backtest must pass both.**

## The five tools

| Tool | Question it answers |
|---|---|
| `check_backtest` | Does your Sharpe survive multiple-testing correction? |
| `noise_sharpe` | What does the *best of N edgeless strategies* score over this sample? |
| `how_long_to_prove` | How many years before this Sharpe is distinguishable from zero? |
| `sample_size_check` | Can your sample even detect the edge you claim? |
| `check_fills` | Are your fills earnable, or better than the market ever offered? |

**The input people omit is the trial count** — how many variants you tried before
settling on this one. Every threshold, lookback and filter you swapped counts. It
is the number that decides the answer, and almost nobody reports it.

## Install

```json
"mcpServers": {
  "backtest-reality-check": {
    "command": "python3",
    "args": ["/absolute/path/to/backtest-reality-check/src/server.py"]
  }
}
```

Pure standard library. No dependencies, no build step, Python 3.8+.

## Try it

```
> check_backtest --sharpe 2.13 --years 1 --trials 34
noise alone at 34 trials  -> Sharpe 2.13
>>> FAILS — your Sharpe is at or below what pure noise produces
```

Thirty-four variants over one year makes a Sharpe of 2.13 the *expected* result of
strategies with no edge whatsoever. That is the whole idea.

Some reference points from `how_long_to_prove`:

| Sharpe | Years of daily data before it's distinguishable from zero |
|---|---|
| 1.5 | 1.2 |
| 1.0 | 2.7 |
| 0.7 | 5.5 |
| 0.5 | 10.8 |
| 0.35 | **22** |

## Verification

The maths is checked against published values, and these are hardcoded as tests:

- MinTRL at SR 1.0 → **2.71 years**
- Max trials for SR 1.0 over 5 years → **45**
- Over 2 years → **7**
- E[max Sharpe] at N=45, 5 years → **1.00**

```
python3 tests/test_anchors.py
```

> A bug I shipped and caught with exactly these: `expected_max_sharpe` first
> returned the raw order-statistic term with **no sample-length scaling**, making
> the bar independent of backtest length so short backtests wrongly passed. Under
> the null the per-observation Sharpe has SD ≈ 1/√T, so the annualised benchmark
> is `max_z / √years`. My own tool failed its own sanity check.

## Sources

Bailey & López de Prado, *The Deflated Sharpe Ratio* (JPM 2014) and *The Sharpe
Ratio Efficient Frontier* (J. Risk 2012); Bailey, Borwein, López de Prado & Zhu,
*Pseudo-Mathematics and Financial Charlatanism* (Notices of the AMS, 2014);
Harvey, Liu & Zhu (RFS 2016) for the multiple-testing hurdle.

## Licence

MIT. Use it, fork it, and please do run it on mine.