Layout
PART III · Inference · CONCEPT 12 OF 24

Power: the chance you’ll notice a real effect.

EXPLORE~8 min
Back to journey
  • Northstar expects a small conversion gain, not a miracle.
  • At a 10% baseline, a 2% relative lift is only 0.2 percentage points.
  • Detecting that reliably can require hundreds of thousands of buyers.
Power, MDE & sample size: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Specify an effect worth finding.
  • The minimum detectable effect (MDE) is a planning effect size tied to a desired power and test.
  • It is not a minimum observed estimate required for significance or a guarantee that smaller effects never appear.

Northstar plans to detect a 2% relative conversion lift from a 10% baseline: only 0.2 percentage points in absolute terms.

Smaller effects cost more.

For this approximate two-sided 5% test at 80% power with equal groups, halving the absolute planning effect requires about four times as many independent buyers.

Halving Northstar's absolute planning effect approximately quadruples required independent buyers under the displayed normal-test assumptions.

Duration is not just traffic divided by n.
  • Use distinct eligible buyers, not requests.
  • Include full business cycles and wait for seven-day outcomes to mature.
  • Low volume, clustering, and exclusions can lengthen the calendar.

Divide by newly eligible distinct Northstar buyers, not repeat requests, then allow business cycles and seven days for included outcomes to mature.

n per arm ≈ 2 × p(1 − p) × (1.96 + 0.842)² / absolute MDE²
LAB 12

Power, MDE & sample size · Northstar

Power at available sample: 18.4%
Absolute planning effect+0.20 pp
Required per arm · 80% power353,209
Minimum collection + follow-up36 + 7 days

Approximate two-sided 5% normal test, 10% baseline, equal allocation, independent binary buyer outcomes. The traffic count is unique newly eligible buyers across both arms, not daily active users counted repeatedly. Add full business cycles, warmup, and operational constraints; this is not a universal sample-size calculator.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

Halving the absolute MDE roughly does what to sample size?

Your learning note

Private · just for you