If you tune a strategy on 2010 to 2026 and report its performance on 2010 to 2026, you have reported how well your search worked, not how well the strategy works. Out-of-sample testing separates those two questions by withholding data from the search.
The standard construction splits the window in time order.
| Segment | Share of window | What it is for |
|---|---|---|
| In-sample | The earlier 70% to 80% | Building rules, tuning parameters, ranking trials |
| Out-of-sample holdout | The final 20% to 30% | One check on whether the winner survives |
A common setup reserves the last 20, 25 or 30 percent of the backtest window, ranks every trial on the earlier in-sample portion only, and then runs the single winning configuration on the reserved segment. If in-sample CAGR was 26% and out-of-sample is 9%, most of the 26% was fitting.
Why the split must be chronological
Randomly sampling dates into training and test sets leaks information, because a strategy trained on March and tested on February is trained on the future. Time-series data has to be split by time. The holdout is always the most recent segment.
That has a side effect worth naming: the holdout is a specific stretch of market history, not a neutral sample. A 2022 to 2026 holdout on Indian equities tests your strategy against a particular rate cycle and a particular small-cap run. Surviving it is evidence, not proof.
What the check can and cannot tell you
It can tell you that a result is not purely an artefact of parameter search. It cannot tell you the strategy will work next year, because the holdout is still history, and it cannot tell you anything about execution, since no orders were sent.
Caveats
The holdout is a single-use resource. Look at it, fail, tweak the rules and look again, and you have simply extended your in-sample period while telling yourself otherwise. If you must iterate, iterate in-sample and accept that the honest out-of-sample sample size shrinks each time you spend it.
Also match the segment to your rebalance frequency. A quarterly strategy tested on a three-year holdout gets 12 rebalances, which is a small enough sample that a single bad quarter dominates the verdict. Longer holdouts buy statistical comfort at the cost of a shorter in-sample window, and there is no setting that escapes that trade.