The backtest trap - why the pattern worked on the slide

Somewhere a slide is being shown to a beginner right now. On it is a line that climbs smoothly from the bottom-left to the top-right, and a caption: "This rule turned ₹1,00,000 into ₹9,40,000 over the last ten years." The line barely dips. The message is unspoken but total — the past has been tested, the machine works, follow it.

The backtest is one of the most persuasive objects in all of trading, because it looks like proof. It is a rule applied to real, historical prices, and it reports a real, historical result. Nothing is made up. And yet a great backtest is one of the easiest things in finance to manufacture, and one of the least likely to survive contact with live money.

So the question for this module is not "does the backtest show a good result?" — it almost always does, or you would not be shown it. The question is: why did the rule work on the slide, and would it have worked for you? Those are different questions, and the gap between them is where beginners lose money.

A backtest answers a narrow, honest question: if this exact rule had been followed over this exact stretch of the past, what would the record show? Used carefully, that is a genuine research tool — a way to discipline an idea before risking a rupee on it. The trouble is that the same tool, used carelessly or dishonestly, becomes a machine for producing impressive numbers that mean nothing at all.

The reason is uncomfortable but simple. A curve that fits the past perfectly usually fits it because it was tuned to it. Give a person enough freedom — which indicator, which threshold, which stocks, which years — and they will find, by search alone, some combination that would have printed money over the chosen history. That combination has not discovered a truth about markets. It has memorised the accidents of one particular past. The slide sells you the fit; it does not sell you an edge, because there may be no edge there to sell.

This is why the module sits here, late in the technicals part, after you have already learned that indicators only re-draw the price and that patterns hide their failures. The backtest is where all of those temptations combine into a single, seductive graphic. In a game this driven by luck, a run of good results tells you almost nothing about whether the method has skill. The whole job of this module is to make you slower to believe a rising line, and to hand you the four questions that separate a test worth respecting from a slide worth ignoring.

Almost every misleading backtest fails in one of five ways, and four of them are traps of how the test was built. Name them once and you will spot them for the rest of your life.

First, data-mining and overfitting. Suppose you try twenty different rules on the same ten years of prices — twenty thresholds, twenty indicator pairs. Purely by chance, one of them will have "worked" beautifully, the way one of twenty coin-flippers will get five heads in a row. Showing you that rule, and quietly discarding the nineteen that failed, is data-mining. The winning rule has overfit — it has bent itself around the random bumps of that decade rather than learning anything that repeats. The more knobs a rule has to turn, the more certain this becomes. A related word for the same disease is curve-fitting: shaping a curve until it hugs the past, which guarantees a good fit and predicts nothing.

Second, no honest holdout. A rule tested on the same prices used to build it is graded against its own answer key — the fit is guaranteed. This is the in-sample trap. The only honest test keeps back a slice of history the rule never saw during design — an out-of-sample window — and checks whether the rule still works there. Surviving fresh data is not proof of an edge, but it is the first evidence that the rule found something real rather than memorising noise. A backtest with no holdout has not been tested at all.

Third, survivorship. If you test a rule only on the stocks that still trade today, you have quietly excluded every company that went to zero, delisted, or was thrown out of the index along the way. Your universe is pre-stocked with winners before the rule does anything. This is survivorship bias, and it flatters almost every strategy, because the graveyard of failed companies — the ones that would have hurt the rule — is simply missing from the data.

Fourth, look-ahead. A subtle, deadly bug: the test uses information that did not exist yet at the moment of the trade. Deciding a morning's buy using that evening's closing price, or a full-year result that was only published months later, lets the rule quietly peek at the future. This is look-ahead bias, and it produces suspiciously smooth curves that can never be reproduced live, because live, you only ever have the past.

The fifth failure is the one everyone forgets, and it deserves its own section: the test ignores what trading actually costs.

Here is the trap that turns a paper winner into a real loser, and it needs only multiplication. illustrative

A backtest reports its result gross — before a single rupee of cost. But a rule that trades reaches that number by making many round-trip trades, and in India every round trip pays real friction. Name them plainly, because "costs" as a vague word is easy to wave away:

• Brokerage — often ₹0 on delivery at a discount broker, but not always, and never on intraday.
• STT (securities transaction tax) — a levy on the value of the trade, both sides.
• Exchange and SEBI fees — small percentage charges the exchange and regulator collect on every order.
• GST — 18% on the brokerage and those exchange fees.
• Stamp duty — a small charge on the buy side.
• DP charges — a flat fee the depository takes on every sell.
• Slippage — the gap between the price you saw and the price you actually got, worst in thin or fast-moving stocks. This is usually the biggest hidden cost of all, and no backtest that fills every trade at the perfect printed price has accounted for it.

Bundle those into one figure — call it the all-in transaction cost per round trip — and watch what it does. Say the slide boasts that ₹100 became ₹250. But the rule churned through sixty round trips to get there, at roughly 1% friction each. Costs compound the same way returns do, just against you: keep 99% sixty times over and you keep about 55% of the value. The ₹250 becomes roughly ₹138. More than two-thirds of the paper gain has quietly evaporated into friction the slide never mentioned.

Every rupee of cost and churn compounds against you exactly as returns compound for you. Nudge the friction to 1.5% — realistic for a small, illiquid stock with heavy slippage — or let the rule trade a hundred times instead of sixty, and the net figure slides below ₹100. The backtest was a winner. You would have been a loser. Nothing about the rule changed; only the honesty of the accounting did.

Numbers on a page are easy to nod along to. Move them yourself and the lesson lands harder. illustrative

The tool below starts on the case from the last section: a slide claiming ₹100 became ₹250. Set how many round-trip trades the rule made and the all-in friction per trade, and watch the gross figure erode into what you would actually have kept. Push the trade count up, or raise the friction to reflect a thinner stock, and see the "winner" fall through ₹100 into a real loss.

The point the tool makes on its own: two backtests can show the identical gross line and hand you completely different outcomes, purely because one traded twice as often or in a stock twice as costly to move. Turnover is not free, and slippage is not optional. A rising equity curve tells you nothing until you know how many times it had to trade to get there.

Step behind the slide and picture how the dazzling rule was actually found. illustrative

A person sits with ten years of prices and tries twenty variations of a rule — twenty moving-average pairs, twenty thresholds, twenty exit levels. Nineteen of them do nothing special, or lose. One of them, by the ordinary luck of trying many things, produces a beautiful climbing line. That one goes on the slide, described as "the system." The nineteen are never mentioned. You are shown the survivor of a search and told it is a discovery.

Now ask the honest question about the winner: is it skilled, or is it lucky? In a search across twenty rules, finding one that shines is exactly what pure chance predicts — the same way one of twenty people flipping coins will hit a long streak and feel gifted. The rule has curve-fit the particular bumps of that decade. Run it on a fresh, unseen stretch of history and the magic almost always fades, because the noise it memorised does not repeat.

This is why the single most useful thing you can ask a backtest is not "what did it return?" but "how many rules did you try before you found this one, and did you test the winner on data it had never seen?" An honest answer — few rules, held-out sample, costs counted — is worth more than any number of percent. An evasive answer — "the chart of the test speaks for itself" — tells you the rule is a survivor of a search, not a finding.

Suppose a backtest clears every bar: few rules, tested out of sample, run on a survivorship-free universe, with full Indian costs and realistic slippage subtracted. That is a serious piece of work, and rare. It still cannot do several things, and pretending otherwise is its own trap.

It cannot recreate the future. Every backtest is a story about conditions that have already happened — a particular decade of rates, flows, and moods. The next decade is not obliged to rhyme, and a rule fitted even honestly to the past carries no guarantee into a market that has never existed before.

It cannot sit in the drawdown for you. A backtest shows the drawdown — the deepest fall from a peak — as a calm number on a chart. Living through it is different. A rule that "worked" over ten years may have spent eighteen months underwater, and almost no one actually holds a losing rule that long with real money on the line. The test assumes a discipline you may not have.

It cannot tell you the rule is worth following even if it is real. Before the vivid story of this one system, the outside view asks how often things like it actually work out. Most rules that survive an honest backtest still earn less, after costs, than simply owning a broad index and doing nothing — which is the quiet, unglamorous base rate that every trading slide is built to make you forget.

The same handful of moves fool beginner after beginner. Name them and the slide loses its spell.

1. Reading the gross line as your result. The climbing curve is before costs. Your line is after brokerage, STT, GST, stamp duty, DP charges, and — above all — slippage. The more it trades, the wider the gap.

2. Accepting an in-sample number as a test. A rule graded on the same prices that built it will always look good. With no held-out, out-of-sample window, nothing has been tested.

3. Ignoring how many rules were tried. One dazzling rule out of twenty is what luck alone produces. Always ask what was discarded before this winner was chosen.

4. Trusting a survivorship-stacked universe. A test run only on stocks that still exist has hidden every company that died. Put the dead names back and most rules shrink.

5. Missing the look-ahead leak. A suspiciously smooth curve often means the test used information that did not exist yet at the moment of the trade. Live, you never have tomorrow's data.

6. Confusing a good fit with a good model. A curve tuned until it hugs the past is guaranteed to fit and likely to fail. The tighter the historical fit, the more suspicious you should be.

7. Forgetting the drawdown is survivable only on paper. The deepest fall is a number in a backtest and a test of nerve in real life. Most people abandon a real rule long before its backtested recovery arrives.