PART IV · Sharper estimates · CONCEPT 13 OF 24
Small tests don’t just miss. Their winners exaggerate.
EXPLORE~7 min
Back to journey- With few buyers and a small true gain, most Northstar tests will be inconclusive.
- The rare significant winners tend to be experiments where noise pushed the estimate unusually high.
| Concept | What it means | Northstar example |
|---|---|---|
| Low power misses useful effects. |
| A small Northstar test can miss an economically useful lift; a wide interval near zero does not establish equivalence. |
| Selection creates the winner’s curse. |
| If Northstar discusses only significant positive cache tests, it preferentially selects estimates that favorable random noise pushed upward. |
| Do not size the next test from the lucky winner. |
| Plan the next Northstar experiment around a worthwhile plausible effect and prior uncertainty, not the unusually large lift of a lucky small-test winner. |
LAB 13
Underpowered tests · Northstar
Significant, positiveSignificant, negative (wrong sign)Not significant
Theoretical power11.6%
Significant runs31 of 300
Exaggeration of significant results3.1×
Significant with the wrong sign0
Exaggeration = mean absolute significant estimate ÷ true effect. In Simulation options, set truth to zero to explore false alarms. Theoretical power and shading use a two-sided 5% known-variance Gaussian approximation at a fixed 10% baseline—not exact binomial inference. Independent runs, not cumulative peeks; seeds are reproducible.
Constructed teaching example · not Amazon data or a production experimentation engine
Why are the rare significant winners in a low-power setting often too large?
Your learning note
Private · just for youFurther reading: Microsoft Experimentation Platform ↗NIST statistical handbook ↗