Layout
PART III · Inference · CONCEPT 11 OF 24

How surprising are the data under no effect?

EXPLORE~7 min
Back to journey
  • A p-value compares the observed evidence with what a specified null model predicts.
  • It is not the probability that the infrastructure change is useless, nor the probability a ship decision is wrong.
P-values & alpha: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Condition on the null.

A two-sided p-value is the probability, assuming the null and test assumptions, of a test statistic at least as extreme in either direction as the observed one.

A Northstar p-value asks how extreme the evidence would be under the specified no-effect model; it does not give the probability the cache is useless.

Alpha controls a long-run error rate.
  • For a valid prespecified test, alpha = 0.05 limits the probability of rejecting a true null to about 5%.
  • It does not say 5% of your significant tests are false discoveries.

A prespecified valid 5% test controls Northstar's rejection probability under a true null, not the fraction of all significant cache tests that must be wrong.

Magnitude still matters.
  • With huge samples, a tiny lift may have a tiny p-value.
  • Report the effect and interval.
  • A p-value cannot price engineering complexity, correctness risk, or operating costs.
  • At enormous Northstar traffic, a commercially negligible lift can be statistically significant.
  • Evaluate its size, uncertainty, cost, and operational risk.
LAB 11

P-values & alpha · Northstar

CONTROL10.00%2,000 / 20,000 buyers
TREATMENT10.20%2,040 / 20,000 buyers
zero effect · -0.39 pp to +0.79 pp
Absolute lift+0.20 pp
Relative lift2.00%
Two-sided p-value0.5069

Hold the observed lift fixed and increase the sample. The interval narrows and the null becomes less compatible with the estimate. This is a sensitivity demonstration, not a sequence of accumulating real outcomes. Interval: unpooled Wald normal approximation; p-value: pooled two-proportion z-test. Independent buyers, mature outcomes, one fixed-horizon analysis; not for sparse or clustered data.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

p = 0.03 means…

Your learning note

Private · just for you