Test one idea and a t-statistic of 2.4 is evidence. Test four hundred and report the best one, and a t of 2.4 is roughly what you should have expected from pure noise. The number did not change. What changed is that it was selected rather than observed.
This is the single largest source of false discoveries in systematic research, and it is easy to fall into without noticing. A scan across 40 signals, 5 forward horizons and 12 sectors is 2,400 tests. Nobody set out to run 2,400 tests. The grid did it silently.
What noise alone produces
Simulated draws from a standard normal, which is what a t-statistic follows when there is no effect at all:
| Independent tests | Expected largest absolute t from pure noise | Chance at least one clears 2.0 |
|---|---|---|
| 1 | 0.8 | 5% |
| 10 | 1.9 | 37% |
| 20 | 2.2 | 61% |
| 50 | 2.5 | 90% |
| 100 | 2.8 | 99% |
| 250 | 3.0 | Above 99.9% |
| 1,000 | 3.4 | Above 99.9% |
Read the second column as a noise bar. If you searched a hundred variations, the best result has to clear roughly 2.8 before it has said anything the search itself would not have produced. Subtracting that bar, rather than comparing against a fixed 2.0, is the correction.
Keeping the count honest
The count that matters is every hypothesis you evaluated, not every one you wrote down. Parameter sweeps count. Abandoned variants count. Ideas tested last month on the same data count, because the data has not been refreshed since. A research log that records each trial as it is run is the only reliable way to know the denominator, and it has to be written before the results arrive.
Correlated tests are not fully independent, so the raw count overstates the penalty. Twenty momentum lookbacks between 60 and 250 days are close to one test, not twenty. The effective number of independent tries is somewhere between one and the raw count, and there is no clean way to pin it down. Err toward the larger number.
Caveats
The Bonferroni correction, which divides the significance threshold by the number of tests, is simple and deliberately harsh. It assumes independence and will discard real effects when the tests are correlated.
No correction rescues a result that was found by search. It only tells you how much of the appearance was search. The genuine test is fresh data the search never touched, either a held-out period or the forward months after you stopped looking.
The deeper problem is that a strategy selected from a large grid inherits the grid’s noise as part of its parameters, which is why the best backtest in a sweep is systematically the one most likely to disappoint out of sample.