You test a momentum rule on the Nifty 500. A 12-month lookback gives 14% CAGR. You try 11 months, then 13, then a 20-name portfolio instead of 30, then a monthly rebalance instead of quarterly. Four parameters with ten settings each is 10,000 combinations, and one of them returns 26%.
That number is not a discovery. With 10,000 draws, a good-looking best result is what you should expect even if none of the parameters matter.
The mechanism
A backtest window contains signal and noise in unknown proportions. Every additional parameter you tune gives the search another way to fit the noise. The reported performance of the winner therefore contains two components you cannot separate by looking at it: whatever real effect exists, plus the luck of being the best of many trials.
The more trials you run, the larger the luck component, and the more the backtest overstates what you will actually earn.
Telling the two apart
| Sign | Likely genuine | Likely overfit |
|---|---|---|
| Parameter surface | A broad plateau where neighbouring settings behave similarly | A single spike, with performance collapsing one step either side |
| Sensitivity | Results degrade smoothly as costs rise | Edge disappears once realistic slippage and STT are applied |
| Reasoning | The mechanism was stated before the test | The explanation was invented after seeing the result |
| Sub-periods | Positive across several regimes | Driven by one year, often 2020 or 2021 |
Defences that actually work
Write the economic reason first, then test it. A rule you can justify before running a backtest has far fewer degrees of freedom than one discovered by search.
Prefer the plateau to the peak. If 11, 12 and 13-month lookbacks all work, the effect is probably real and you should pick the middle. If only 12 works, you have found an artefact.
Count your trials honestly. Optimisation on saral.money runs a parameter sweep as an explicit grid with heatmap and scatter views over the trial set, which makes the count visible instead of leaving it in your head as “I tried a few things”.
Reserve a segment of history you do not look at until the end. That is out-of-sample testing, and it is the only check that directly measures the luck component.
Caveats
Out-of-sample testing reduces the problem rather than removing it. If you run a holdout, fail, adjust the strategy and run the same holdout again, the reserved data has become part of your search. You get one honest look at it.