A p-value answers a narrow and slightly awkward question. Assume the effect you are testing does not exist. How often would random data alone hand you a result this large or larger?
A p of 0.03 says three times in a hundred. That is a statement about the data under an assumption of nothing, and it is worth being precise about what it does not say. It is not the probability that the assumption of nothing is true, and it is not the probability that the strategy will make money. Those are different quantities and the p-value has no access to them.
For a mean estimate the p-value is a direct translation of the t-statistic:
p = 2 x (1 - CDF(|t|))
| Absolute t | Two-sided p |
|---|---|
| 1.00 | 0.317 |
| 1.65 | 0.099 |
| 1.96 | 0.050 |
| 2.00 | 0.046 |
| 2.58 | 0.010 |
| 3.00 | 0.003 |
The two columns carry identical information. A screen showing both is showing one number twice.
Reading it in a backtest
Suppose a seasonality scan over 15 years of Nifty data reports that the average return in a particular calendar month was +2.1% with a p-value of 0.04. Fifteen years gives you fifteen observations of that month. The p-value is computed from those fifteen numbers, and it is telling you that fifteen draws this favourable would arise by chance in about one sample in twenty-five. Given that a calendar scan tests twelve months, and probably several universes alongside them, one result at 0.04 is what the scan produces when nothing is there.
Caveats
Any effect becomes significant with enough observations. A p-value shrinks as n grows regardless of whether the effect is large enough to survive brokerage, impact and taxes. Read the effect size first and the p-value second. A statistically significant +0.04% per rebalance is not tradeable at Indian smallcap spreads.
The conventional 0.05 threshold is a convention with no special standing. Treating 0.049 and 0.051 as different categories of truth is the most common misuse of the number.
The p-value assumes a single pre-specified test on independent observations. Both assumptions break in ordinary strategy research. Overlapping windows violate independence, and any scan violates the single-test assumption, at which point the reported p-value understates the true false-positive rate by orders of magnitude.
A large p-value is not proof of absence. It usually means the sample is too small to resolve the question either way.