process
The backtest illusion
The rule
A rule tuned until it fits the past perfectly tells you about the past, not the future. Squeeze the data long enough and it will agree to anything.
Where it flips
Not every backtest is useless. A simple idea with few knobs, tested on data it never saw, can carry real information. The sin is over-tuning, not checking history at all.
Mauboussin warns: do not mistake a good fit for a good model. If you test enough settings against old prices, some mix will have 'worked' beautifully. But that is overfitting. The rule has only memorised the noise of one past, not learned anything that repeats. The more knobs you turn (which RSI level, which pair of moving averages), the more sure you are to find a curve that fits and then fails with real money. When a strategy shows a dazzling backtest, be suspicious. The demo was built by searching the very history it is now being sold on.
A worked example
A course sells a system that 'returned 400% in backtests'. It was found by trying thousands of setting mixes on one decade of data. The winner fit that decade's noise and fell apart the moment real money followed it. [illustrative]
How to spot it
- ·few knobs, not many, behind the rule
- ·a slice of data held back and tested on unseen data
- ·you stay suspicious of any strategy sold on its own dazzling backtest
Michael Mauboussin · Think Twice