Layout
PART IV · Sharper estimates · CONCEPT 13 OF 24

Small tests don’t just miss. Their winners exaggerate.

EXPLORE~7 min
Back to journey
  • With few buyers and a small true gain, most Northstar tests will be inconclusive.
  • The rare significant winners tend to be experiments where noise pushed the estimate unusually high.
Underpowered tests: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Low power misses useful effects.
  • An insignificant result from a small test can coexist with a wide interval containing meaningful gains and harms.
  • It does not establish equivalence.

A small Northstar test can miss an economically useful lift; a wide interval near zero does not establish equivalence.

Selection creates the winner’s curse.
  • If you discuss only significant positive tests, you select estimates helped by positive noise.
  • Their average lift can exceed the true effect even when the estimator is unbiased before selection.

If Northstar discusses only significant positive cache tests, it preferentially selects estimates that favorable random noise pushed upward.

Do not size the next test from the lucky winner.
  • Plan using a worthwhile plausible effect, prior evidence, and uncertainty.
  • Replication and shrinkage can help; repeatedly running small tests until one wins does not solve the problem.

Plan the next Northstar experiment around a worthwhile plausible effect and prior uncertainty, not the unusually large lift of a lucky small-test winner.

LAB 13

Underpowered tests · Northstar

Each dot is one independent simulated A/B test; green is significant positive, red is significant negative, gray is not significant.ESTIMATED EFFECT (PERCENTAGE POINTS)-4.0-2.00.0+2.0+4.0+6.0Not significantTrue effect +1.00 ppAverage significant estimate +3.11 pp300 INDEPENDENT RUNS OF THE SAME TEST
Significant, positiveSignificant, negative (wrong sign)Not significant
Theoretical power11.6%
Significant runs31 of 300
Exaggeration of significant results3.1×
Significant with the wrong sign0
Simulation options

Exaggeration = mean absolute significant estimate ÷ true effect. In Simulation options, set truth to zero to explore false alarms. Theoretical power and shading use a two-sided 5% known-variance Gaussian approximation at a fixed 10% baseline—not exact binomial inference. Independent runs, not cumulative peeks; seeds are reproducible.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

Why are the rare significant winners in a low-power setting often too large?

Your learning note

Private · just for you