How surprising are the data under no effect?
- A p-value compares the observed evidence with what a specified null model predicts.
- It is not the probability that the infrastructure change is useless, nor the probability a ship decision is wrong.
| Concept | What it means | Northstar example |
|---|---|---|
| Condition on the null. | A two-sided p-value is the probability, assuming the null and test assumptions, of a test statistic at least as extreme in either direction as the observed one. | A Northstar p-value asks how extreme the evidence would be under the specified no-effect model; it does not give the probability the cache is useless. |
| Alpha controls a long-run error rate. |
| A prespecified valid 5% test controls Northstar's rejection probability under a true null, not the fraction of all significant cache tests that must be wrong. |
| Magnitude still matters. |
|
|
P-values & alpha · Northstar
Hold the observed lift fixed and increase the sample. The interval narrows and the null becomes less compatible with the estimate. This is a sensitivity demonstration, not a sequence of accumulating real outcomes. Interval: unpooled Wald normal approximation; p-value: pooled two-proportion z-test. Independent buyers, mature outcomes, one fixed-horizon analysis; not for sparse or clustered data.