A backtest from January 2010 to December 2025 covers roughly 3,950 NSE trading sessions. Reserve the last 25% and about 990 of them, just under four years, become data your selection process is not allowed to look at.
Those 990 days are the only part of the run capable of telling you something you did not build in yourself. Every metric computed on the other 2,960 days is, to some degree, a description of choices you made after seeing them.
That is the entire argument for out of sample testing, and it is worth being precise about both what it buys you and what it does not.
The split, stated properly
In-sample is the segment your selection process is allowed to see. You fit parameters on it, rank trials on it, look at equity curves on it, and change your mind on it as often as you like.
Out-of-sample is the reserved segment. Selection never sees it. Not the ranking, not the eyeballing, not the “let me just check whether the drawdown looks reasonable”. It is consulted once, after the choice is locked, and it reports whether the parameters you picked survived data they were never picked on.
The word “never” is doing real work there. A holdout that influenced a decision, even a small one, has become in-sample data with a misleading label. There is no partial credit and no way to un-see a result.
How it works mechanically
Take a 12-month momentum screen on the Nifty Midcap 150. You rank by @ROCP(close, n), sweep the lookback n over {126, 189, 252} and the number of holdings over {15, 20, 25, 30}, and rebalance monthly. That is 12 trials.
Set the window to January 2010 through December 2025 with a 25% holdout. The split lands around the start of 2022. Everything from 2010 to 2021 is in-sample; 2022 onward is reserved.
The sequence is then:
- Run all 12 trials over the full window.
- Compute metrics separately on each side of the split date.
- Rank the 12 trials using the in-sample metrics only. The winner is chosen without the reserved segment casting a vote.
- Report the winner’s out-of-sample metrics beside its in-sample metrics.
The number you read is retention: how much of the in-sample edge showed up afterwards. Using illustrative figures rather than a measured run, if the winner posted an in-sample Sharpe of 1.30 and an out-of-sample Sharpe of 0.78, retention is 60%. If in-sample CAGR was 21% and out-of-sample CAGR was 11%, retention on that axis is 52%.
Retention around 50% to 70% on a strategy with a real mechanism is unremarkable and roughly what you should expect, because the in-sample number was selected as a maximum and the out-of-sample number was not. Retention near zero says the parameters described the past rather than the market. Retention above 100% is not a triumph, it is a reminder that both numbers are noisy estimates.
Parameters that change the answer
Holdout size is the first choice. A 20% holdout leaves more history to select on and less to validate with; a 30% holdout does the reverse. Neither is correct in general, and the trade is genuinely two-sided: a longer in-sample segment gives the ranking more regimes to chew on, while a longer holdout makes the validation number less noisy.
Where you cut is not really a choice. The holdout must be the tail of the window, not a random slice or a middle section. Financial returns cluster in time, in volatility and in regime, so a randomly sampled holdout leaks: the model sees March 2020 in-sample and is validated on April 2020, which is not out-of-sample in any meaningful sense. Taking the tail preserves the one property you care about, that the validation data comes after the selection data, which is the only ordering live trading will ever give you.
There is a subtler leak at the boundary. The last in-sample observation and the first out-of-sample observation are adjacent days, and any signal computed over a lookback window spans both. López de Prado’s purging and embargo techniques address this by dropping a buffer around the split. For a monthly-rebalanced strategy with a 252-day lookback the effect is small relative to a four-year holdout, but it is not zero, and it grows as the holdout shrinks.
Where a holdout stops working
Five failure modes, in rough order of how often they bite.
The first is reuse. You look at the out-of-sample result, decide the lookback should have been 189 rather than 252, re-run, and look again. The holdout is now part of your selection process and the second reading means nothing. This failure is a bookkeeping problem rather than a technical one, which is why it is so common: nothing in the software stops you, and the temptation arrives precisely when the first answer is disappointing.
The second is that one holdout is one draw. A reserved segment covering 2022 to 2025 measures your strategy against one macro regime, and a different four-year window would have produced a different number. Lo (2002), “The Statistics of Sharpe Ratios” in the Financial Analysts Journal, gives the rough standard error of an annualised Sharpe under independent returns as SE = sqrt((1 + SR^2 / 2) / T), with T in years. An out-of-sample Sharpe of 0.78 measured over four years therefore carries a standard error of about 0.57. The approximate 95% interval runs from roughly -0.34 to 1.90. That interval contains zero, and it also contains a very good strategy. A single holdout narrows the range of things you should believe. It does not pin down a number.
The third is that a short holdout is worse than no holdout, because it looks like evidence. The same formula at one year of reserved data gives a standard error above 1.2 for the same Sharpe. Below roughly 30 reserved days the exercise is not measuring anything at all.
The fourth is specific to how a selection holdout is built. Both segments come from a single backtest run, so the portfolio at the split date was assembled in-sample. If the book held a midcap name such as FEDERALBNK on 31 December 2021, that position was established by in-sample logic and carries into the out-of-sample segment along with its cost basis. This is a true selection holdout, and it is not a full walk-forward re-fit where the model is retrained from scratch at each step.
The fifth is scope. A holdout tests the selection decision, and nothing else. Survivorship bias, look-ahead in a fundamental field, optimistic slippage assumptions and unfillable trades all pass straight through the split untouched, because they contaminate both segments equally. A strategy can survive a holdout beautifully and still be untradeable.
Why walk-forward is the stronger idea
A single holdout produces one out-of-sample observation. Walk-forward produces many.
The concept: fit on 2010 to 2014, test on 2015, roll forward, fit on 2011 to 2015, test on 2016, and so on. Every year after the first training window becomes an out-of-sample observation, and the parameters are refreshed at each step exactly as you would refresh them in live trading. Instead of a single retention figure you get a distribution of them, and the distribution is what tells you whether the edge is stable or whether it worked once.
Walk-forward is on the saral.money roadmap and is not available yet. The single out-of-sample holdout described below has shipped. Until walk-forward arrives, the practical substitute is the combinatorial approach described in the companion post on backtest overfitting: CSCV runs thousands of in-sample and out-of-sample splits over the returns a sweep already produced, which gives you the distribution a single holdout cannot. It is not a re-fit, so it is not a replacement for walk-forward. It is a much better answer than one draw.
How saral.money handles the holdout
The backtest dialog offers an out-of-sample holdout of None, 20%, 25% or 30%, defaulting to 25%. When a holdout is active, trials are ranked on the earlier in-sample segment only, and the reserved segment appears as a separate column reporting whether the winning parameters survived data they were never selected on. The metrics shown in the main trial table are the in-sample values, labelled as such, so the ranking number and the validation number are never confused for each other. When the window is too short for the reserved tail to mean anything, under roughly 30 reserved days, the run proceeds without a holdout rather than reporting noise as validation. You can see the rest of the backtesting and optimisation surface on the features page, and the strategy templates are a reasonable place to try a split on something already built.
Further reading
For the mechanics underneath a holdout, see the step-by-step backtesting guide, which covers survivorship bias, look-ahead and cost assumptions. The selection-bias statistics that complement a holdout are in backtest overfitting and the deflated Sharpe test.
Glossary terms worth having straight: out-of-sample testing, backtesting, the Sharpe ratio and max drawdown.
Lo, A. W. (2002), “The Statistics of Sharpe Ratios”, Financial Analysts Journal 58(4), is the short paper to read if you want to know how much confidence a Sharpe ratio deserves. It is about eight pages and it will change how you read every performance table you see afterwards.
What to try next: take a strategy you already trust, set a 25% holdout, and write down your prediction for the out-of-sample Sharpe before you run it. Then run it once. The gap between your prediction and the result is a more honest measure of how well you understand your own strategy than the result itself.