Backtesting and the overfitting trap

Someone shows you a trading rule. On the chart it is beautiful. Every arrow points the right way; every winner is circled; the account curve climbs from the bottom-left to the top-right like a staircase to heaven. "This rule made 40% a year for the last five years," they say. "Backtested. The numbers don't lie."

The numbers, in fact, lie constantly — not by being fake, but by being true about the wrong thing. A backtest is the exercise of running a trading rule over past price data to see how it would have done. It is a genuinely useful tool. It is also the single easiest place in all of technical analysis to fool yourself, because the past is fixed and known, and anything fixed and known can be fitted perfectly by someone determined enough to try.

The uncomfortable question this module answers is the one that should nag at you every time you see a glorious backtest: if this rule really made 40% a year, why is the person selling it to you for ₹2,000 instead of quietly running it? The honest answer, most of the time, is that the rule worked wonderfully on the exact history it was built from, and nowhere else.

Testing an idea against history is not the problem. Done carefully, it is one of the few honest checks a chart-reader has. The problem is that the past holds still while you experiment on it, and a person experimenting on a fixed target will, sooner or later, hit the bullseye by luck alone.

Picture searching a long price history for "the rule that would have worked." You try a 50-day high; not bad. You add a volume filter; better. You skip Mondays because they dragged the numbers; better still. You require the 20-day average above the 50-day; better again. Each tweak improves the past because you chose it to improve the past. Twenty tweaks later you have a rule that turned ₹1 lakh into ₹1 crore on paper — and has learned nothing except the specific, random shape of those particular five years.

This is overfitting, also called curve-fitting: tuning a strategy so tightly to old data that it captures the noise of that history rather than any real, repeatable pattern. The tell is that the rule has many conditions, each one added because it flattered the past. A rule with two conditions has little room to memorise noise. A rule with eleven conditions has enormous room, and almost always has.

There is a plain statistical reason this is inevitable, not merely possible. If you test enough rules against one history, some will look brilliant purely by chance — the way that, in a room of four thousand coin-flippers, several will flip five heads in a row and look like prophets. This is data mining: search a large enough space of possible rules and you are guaranteed to find winners that mean nothing. The winning rule is not a discovery. It is the survivor of a lottery you ran without noticing.

That is why this module exists. A backtest feels like proof — hard numbers, real history, no opinions. But a number that came from fitting and scoring on the same fixed past is not evidence about the future. It is a mirror, and it shows you exactly what you went looking for.

Overfitting is the headline trap, but it travels with two quieter companions. Together, these three do almost all the damage, and each one makes a losing idea look like a winning one.

Overfitting (curve-fitting). Covered above: too many conditions, each chosen to improve the past, until the rule has memorised noise. The paper curve is spectacular; the live curve is flat or worse. This is the difference between an edge and a coincidence dressed as one — a fitted rule is a coincidence you paid to discover.

Survivorship bias. This one hides in the data itself, before a single rule is written. Suppose you test a strategy on "the current Nifty 500." That list contains only companies healthy enough to still be listed today. Every company that collapsed, got delisted, or went to zero in those years is simply absent. You have tested your rule in a universe where failure was quietly deleted in advance — so of course "buy the breakout and hold" looks wonderful. In the real world you would have bought some of the companies that later vanished. Survivorship bias is judging a strategy only on the winners that remain, and it inflates almost every naive backtest.

Look-ahead bias. The subtlest of the three. It means letting the backtest use information it could not have known at the moment of the trade. You "buy on the day of the results" — but the results came out after the close, so at the moment you claim to buy, that price did not yet exist. Or you rank stocks by a full-year figure that was only reported months later. Look-ahead bias lets tomorrow's information leak into today's decision, and even a tiny leak can turn a losing rule into a paper champion.

Set the three side by side and the shape is always the same: the backtest lives in a kinder world than the market does.

The defence against all three has one name: the out-of-sample test. You tune your rule on one slice of history, then check it — untouched, no more tweaking — on a different slice it has never seen. If the edge survives the fresh data, it might be real. If it evaporates, you were fitting noise. The paper curve and the fresh-data curve almost never look alike, and the gap between them is the size of the lie.

Watch the trap close on an ordinary, well-meaning reader. illustrative

She reads about a breakout rule and decides to test it herself — good instinct. She downloads five years of daily data for a basket of well-known stocks and codes a rule: buy when price makes a new 50-day high on above-average volume, sell after a 12% gain or an 8% loss. The first run is only okay. So she starts improving it. She adds a filter to skip trades when the broader market is below its 200-day average. Better. She raises the volume threshold. Better. She adds a rule to avoid the two weeks around results season, because those trades were messy. Better again. After an afternoon of tuning, the paper account is up 38% a year with a shallow, tidy drawdown.

Every step felt like learning. It was actually the opposite. Each filter was chosen because it improved the same five years she was scoring against — so the rule slowly memorised the accidents of that particular history. Two hidden flatterers helped: her stock basket was today's list of survivors, so it excluded the companies that quietly died over those years, and her "buy at the new high" quietly used the day's high price, information not fully known until the day was over.

She funds a real account. Over the next eight months the rule takes twenty trades. It is not a disaster — it is worse than that, because a disaster would be obvious. It is a slow bleed: a few small wins, several 8% losses, costs and slippage nibbling every trade, the account drifting down while the paper version she still admires would have been up 25%. Nothing dramatic broke. The rule simply never had the edge the backtest showed; the backtest had shown her the shape of the past, and the past does not repeat on command. This is a rule that fit the history perfectly and forecast nothing.

The repair was never a better filter. It was one discipline she skipped: tune on the first three years, then test — once, untouched — on the last two. Had she done that, the 38% would have collapsed to something honest on the fresh slice, and she would have known before funding it that she was holding a mirror, not an edge.

Even a clean, honest backtest — out-of-sample, survivor-inclusive, no leaks — has hard limits, and pretending otherwise is its own trap.

It cannot tell you the future will resemble the past. A rule that genuinely worked for a decade can stop working the moment enough people find it, or the market regime shifts. Backtests are silent about change; they can only describe the world that already happened.

It cannot capture the rare, violent day. Most of a strategy's fate is decided by a handful of extreme moves, and those are exactly what a smooth backtest under-weights. Real markets have far more extreme days than a tidy model expects, and a single gap through your stop can erase a year of small, backtested wins. The paper curve rarely shows that day; the live account eventually meets it.

It cannot include the frictions that live trading forces on you — the real spread, the slippage when your order walks the book, the tax, the brokerage, and above all the human being who abandons the rule after four losses in a row. A backtest assumes a disciplined robot with zero costs. You are neither.

And it cannot rescue you from the deepest point of this whole shelf: even a perfect edge on paper is not a licence to stop thinking. There is no rule so well-tested that it removes the need for judgement and risk control. The backtest is a question, not an answer.

The same handful of moves catch tester after tester, and seller after seller. Named once, they are far easier to catch.

1. Scoring a rule on the data it was tuned on. The commonest error of all. If the rule was improved on those five years, its performance on those five years is meaningless. Only fresh data counts.

2. Admiring a strategy with many conditions. Each extra filter feels like sophistication and is usually just another knob turned to flatter the past. Distrust complexity; the more rules, the more likely you are looking at fitted noise.

3. Testing on today's survivors. Any list of "current" stocks has already deleted the failures. A backtest on survivors is a test run in a world where losing was made impossible.

4. Believing a smooth equity curve. Real edges are lumpy and painful; a suspiciously smooth backtest usually means a leak, a fit, or a missing extreme day, not a superior rule.

5. Trusting a number with no costs in it. Remove spread, slippage, and tax and almost any active rule looks profitable. The costs are where most paper edges quietly die.