Layout
PART III · Inference · CONCEPT 9 OF 24

A flat truth can produce a lively dashboard.

EXPLORE~5 min
Back to journey
  • Imagine the infrastructure change has exactly zero conversion effect.
  • Repeating the experiment with different buyers will still produce positive and negative estimated lifts.
Sampling variation: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
The estimate is random.
  • Buyer outcomes vary even when the underlying conversion probabilities are fixed.
  • A single observed lift combines the true effect and sampling error.

Even if Northstar's cache has no true conversion effect, different random buyer samples can produce positive or negative observed lifts.

More buyers shrink noise.
  • For independent, similarly distributed observations, standard error scales roughly as 1/√n.
  • Four times the sample gives about half the standard error, not one quarter.

With the independent-buyer approximation, quadrupling Northstar's sample roughly halves standard error—not quarters it.

A/A is not always exactly zero.
  • Under a valid 5% test, false positives are expected about 5% of the time under the null.
  • Across many A/As, calibration is more informative than one surprising result.
  • Across many well-calibrated Northstar A/A tests, some ordinary 5% tests should still reject a true null.
  • Evaluate the procedure across repetitions.
LAB 09

Sampling variation · Northstar

Each dot is one independent simulated A/B test; green is significant positive, red is significant negative, gray is not significant.ESTIMATED EFFECT (PERCENTAGE POINTS)-4.0-2.00.0+2.0+4.0+6.0Not significantTrue effect +1.00 ppAverage significant estimate +3.11 pp300 INDEPENDENT RUNS OF THE SAME TEST
Significant, positiveSignificant, negative (wrong sign)Not significant
Theoretical power11.6%
Significant runs31 of 300
Exaggeration of significant results3.1×
Significant with the wrong sign0
Simulation options

Exaggeration = mean absolute significant estimate ÷ true effect. In Simulation options, set truth to zero to explore false alarms. Theoretical power and shading use a two-sided 5% known-variance Gaussian approximation at a fixed 10% baseline—not exact binomial inference. Independent runs, not cumulative peeks; seeds are reproducible.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

With no true effect, can a valid test show a positive lift?

Your learning note

Private · just for you