randomness
The backtest trap
The rule
A rule that 'always worked' on the old chart is usually a trick of picking winners and fitting too tight. The past describes; it does not promise.
Where it flips
This does not mean all history is useless. A rule tested honestly on data it never saw can still carry a modest edge. The fix is to tell a fitted story apart from a rule checked on new, unseen data, and to distrust any signal shown only through its winners.
Taleb warned that with enough tries you can always find a rule that fits old data perfectly. That rule tells you about the past, not the future. A setup shown only on the charts where it worked is picking winners. A rule tuned until it caught every old turn is fitting too tight. Both feel like a discovery, and both tend to fall apart once real money and new data arrive. The many failed tries you never saw are what the confident test quietly hides.
A worked example
A course shows five charts where a moving-average crossover caught the exact bottom. It looks perfect. Used forward on fresh data across many stocks, the same rule whips back and forth and gives back its gains. The five examples were just the survivors, picked afterward from thousands. [illustrative]
How to spot it
- ·a rule shown only through hand-picked charts where it worked
- ·settings tuned until the setup caught every past turn
- ·no test on fresh or unseen data, and no mention of the failures
Nassim Nicholas Taleb · Fooled by Randomness