Backtesting

Backtest Overfitting: The Deflated Sharpe Test

Saral Team · 1 August 2026

Fifty parameter combinations. The best one comes back with a Sharpe of 1.8, and every instinct says you have found something.

Now run the same sweep on fifty strategies with no edge at all. Pure coin flips. The best of those fifty will also look good, because selecting the maximum of fifty noisy numbers is not the same as measuring one number. The maximum of a sample is biased upward by construction, and the bias grows with how many things you tried.

That gap between “the best result I saw” and “the best result noise would have handed me anyway” is what backtest overfitting actually is. There are two published statistics that measure it, and both are cheap to compute once you have already run the trials.

What backtest overfitting means precisely

Overfitting in a parameter sweep is not the same thing as a model with too many free parameters. The mechanism is selection.

Each trial in a sweep produces a Sharpe ratio that is an estimate, carrying sampling error. You then pick the largest estimate and report it as though it were a measurement. The reported number therefore contains two things you cannot separate by looking at it: whatever true edge exists, plus the largest positive error in the batch.

The fix is to ask what the largest error alone would look like, and require the observed winner to clear that.

The expected maximum Sharpe of N skill-less trials

Bailey and López de Prado (2014), in “The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality” (Journal of Portfolio Management 40(5)), give the benchmark directly. Under a null where every trial has zero true Sharpe, the expected maximum across N trials is approximately

E[max SR] = sqrt(V) * [ (1 - g) * Z(1 - 1/N) + g * Z(1 - 1/(N*e)) ]

where V is the variance of the Sharpe ratios across trials, Z is the inverse standard-normal CDF, g is the Euler-Mascheroni constant 0.5772, and e is Euler’s number.

The shape of that expression matters more than the algebra. The bar scales with the spread of your trial results and rises with the number of trials, but it rises slowly, as the tail of a normal distribution does. Evaluating it gives the multiplier on the cross-trial standard deviation:

Combinations tried	Noise bar, in cross-trial standard deviations
4	1.05
8	1.46
16	1.80
25	2.00
50	2.28

Going from 8 combinations to 50 raises the bar by about 56%. Not by 6x, which is what a naive Bonferroni instinct might suggest, but not by nothing either.

A worked example on a 25-combination sweep

Take an RSI mean-reversion screen on the Nifty 100. The entry rule is @RSI(close, n) < t, and you sweep the lookback n over {7, 10, 14, 21, 28} and the threshold t over {20, 25, 30, 35, 40}. That is 25 combinations, comfortably inside a 50-trial cap.

Suppose the 25 annualised Sharpes come out with a mean of 0.55, a cross-trial standard deviation of 0.40, and a best of 1.35, over roughly 1,500 trading days. The arithmetic below is worked from the published formula with those assumed inputs. It is an illustration of the mechanics, not a measured backtest.

The noise bar at 25 trials is 2.00 * 0.40 = 0.80 annualised. The observed 1.35 clears it. Good so far.

The Deflated Sharpe Ratio then converts that margin into a probability:

DSR = Z_cdf( (SR - SR*) * sqrt(T - 1) / sqrt(1 - g3*SR + (g4 - 1)/4 * SR^2) )

Here SR and SR* are in per-period units, T is the number of observations, g3 is the skewness of the winning trial’s returns and g4 its kurtosis. With T = 1500, a skewness of -0.5 and a kurtosis of 6, this comes out at roughly 0.90.

Which is the interesting part. The winner beat the luck bar and still did not reach the conventional 0.95 threshold. Now hold everything else fixed and change only how many combinations you tried:

Combinations tried	Noise bar (annualised Sharpe)	Deflated Sharpe
8	0.58	0.97
25	0.80	0.90
50	0.91	0.85

Same strategy, same 1,500 days, same observed Sharpe of 1.35. The only thing that changed is the size of the search. The evidence weakens as the search widens, which is exactly the property you want and exactly the property a plain Sharpe number does not have.

One practical note from that arithmetic. At daily frequency the per-period Sharpe is small, around 0.085 in this example, so the skewness and kurtosis correction moves the answer by about one percentage point. The number of trials does the heavy lifting. Non-normality matters far more for a strategy with a fat left tail measured over a short window, which is a reason to be suspicious of short backtests on illiquid names rather than a reason to ignore the term.

PBO answers a blunter question

The Deflated Sharpe asks whether the winner’s number is too big to be luck. The Probability of Backtest Overfitting asks something more directly useful: when you pick the in-sample winner, how often does that pick turn out to be a below-median performer out of sample.

Bailey, Borwein, López de Prado and Zhu (2015), “The Probability of Backtest Overfitting” (Journal of Computational Finance), estimate it with combinatorially symmetric cross-validation, or CSCV. The procedure runs on the returns you already stored:

Build a T x N matrix of per-period returns, one column per trial. A grid sweep produces exactly this shape for free.
Slice the rows into S contiguous, equal blocks. The paper uses S = 16.
For each of the C(S, S/2) ways to pick half the blocks as in-sample, with the complement as out-of-sample: find the trial with the best in-sample Sharpe, then find that same trial’s rank among all N trials out of sample.
PBO is the fraction of splits where the in-sample winner landed at or below the out-of-sample median.

With S = 16 that is C(16, 8) = 12,870 distinct splits. If the in-sample winner falls below the out-of-sample median in 7,850 of them, PBO is 0.61.

Read it against 0.5. A selection procedure that carries no information scores 0.5, because the winner’s out-of-sample rank is a coin flip. Below 0.2 the ranking is genuinely predictive. Above 0.5 the ranking is actively misleading: the thing that looked best in-sample is more likely than not to be a laggard afterwards, which happens when the parameters are fitting the specific sequence of returns rather than a repeatable relationship.

“Combinatorially symmetric” is the load-bearing word. Every block serves as in-sample in exactly half the splits and out-of-sample in the other half, so the estimate does not depend on where one train-test boundary happened to fall. It also costs no additional backtests. Every number comes from post-processing return series you already have, which is what makes it practical to run on every sweep rather than as a special exercise.

CSCV reports two useful side quantities as well. The slope of out-of-sample Sharpe on in-sample Sharpe across splits tells you how much of the in-sample edge survives; a slope at or below zero means a better in-sample fit predicted a worse result afterwards. And the share of splits where the selected trial lost money out of sample is a blunt, readable risk number.

Where both statistics fail

Neither of these is a certificate. Four failure modes are worth naming plainly.

The honest N is every configuration you ever looked at, not the size of the final grid. If you hand-tuned six versions before setting up the sweep, then discarded a seventh because the drawdown looked ugly, the search was 56 wide and not 50. Nothing in the formula can know that. This is the same discipline problem covered in cognitive biases that destroy algo strategies, and it is the one that bites hardest, because the under-count is always in the flattering direction.

The cross-trial variance can be gamed without meaning to. A grid whose values are all nearly identical produces a small V, a low bar, and a flattering Deflated Sharpe. A deliberately wide grid produces the opposite. The statistic is measuring the spread of the pool you chose to search, so a grid designed around parameters you already believe in will understate the correction.

Trials are not independent. An RSI lookback of 13 and one of 14 produce almost the same return series, so 50 correlated trials are nowhere near 50 independent tests. Both statistics treat the trial set as a selection pool and lean on the cross-trial spread to capture its diversity, which is a reasonable approximation and not a guarantee.

And neither one touches data quality. A DSR of 0.99 on a backtest that quietly assumes you could buy HINDALCO at the closing print, or that trades a smallcap sitting under an ASM surveillance framework at a price nobody could get, is a precise measurement of a fiction. Selection-bias correction sits on top of a clean backtest. It does not create one. The backtesting fundamentals guide covers what has to be right underneath: survivorship, look-ahead, costs and fill assumptions.

There is also a limit built into the question itself. A DSR of 0.96 is a failure to reject a null, not proof of an edge. It means the result is unlikely to be a pure artefact of the search you ran. Everything else, whether the mechanism makes economic sense, whether the trades were fillable, whether the regime persists, remains your job.

How saral.money reports this

The optimisation view shows a Deflated Sharpe figure and an Overfitting risk figure beside the trial table, so the correction sits next to the number it corrects rather than in a separate report you have to go and ask for. Both are computed from the returns the sweep already produced. When there is not enough to work with, they show as unavailable rather than as a reassuring number: the Deflated Sharpe needs at least 2 completed trials, and the overfitting figure needs at least 4 trials and roughly 40 days of returns before CSCV can build meaningful blocks. Parameter sweeps are capped at 50 combinations, which is a deliberate limit rather than a technical one. You can also read the trial set directly through the heatmap and scatter views, then adopt a winning parameter set into a strategy variation or narrow the grid and re-run. The features page covers the rest of the optimisation surface.