PART III · Inference · CONCEPT 9 OF 24
A flat truth can produce a lively dashboard.
EXPLORE~5 min
Back to journey- Imagine the infrastructure change has exactly zero conversion effect.
- Repeating the experiment with different buyers will still produce positive and negative estimated lifts.
| Concept | What it means | Northstar example |
|---|---|---|
| The estimate is random. |
| Even if Northstar's cache has no true conversion effect, different random buyer samples can produce positive or negative observed lifts. |
| More buyers shrink noise. |
| With the independent-buyer approximation, quadrupling Northstar's sample roughly halves standard error—not quarters it. |
| A/A is not always exactly zero. |
|
|
LAB 09
Sampling variation · Northstar
Significant, positiveSignificant, negative (wrong sign)Not significant
Theoretical power11.6%
Significant runs31 of 300
Exaggeration of significant results3.1×
Significant with the wrong sign0
Exaggeration = mean absolute significant estimate ÷ true effect. In Simulation options, set truth to zero to explore false alarms. Theoretical power and shading use a two-sided 5% known-variance Gaussian approximation at a fixed 10% baseline—not exact binomial inference. Independent runs, not cumulative peeks; seeds are reproducible.
Constructed teaching example · not Amazon data or a production experimentation engine
With no true effect, can a valid test show a positive lift?
Your learning note
Private · just for youFurther reading: Microsoft Experimentation Platform ↗NIST statistical handbook ↗