An interval from −0.6 to +1.0 points supports a very different decision from one spanning +0.1 to +0.3 points.
Confidence intervals: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
Concept
What it means
Northstar example
↔️Interpret the procedure.
A 95% confidence procedure covers the fixed true effect in about 95% of repeated experiments under its assumptions.
It does not assign a 95% probability to this realized interval containing that fixed effect.
🛒 Across repeated appropriate experiments, Northstar's 95% interval procedure should cover the fixed true effect about 95% of the time.
🎯Compare with zero and with worthwhile effects.
An interval can exclude zero but include only commercially tiny effects.
Another can include zero while still permitting an important gain or loss.
‘Not significant’ does not mean ‘no effect’.
🛒 Northstar distinguishes a tiny, precise conversion improvement from a wide interval that still permits both meaningful benefit and harm.
📐Check the approximation.
The playground uses a large-sample normal interval for a difference of independent buyer conversion proportions.
Sparse events, clusters, sequential looks, and selection require different treatment.
🛒 The normal-proportion illustration is not automatically suitable for Northstar's sparse errors, clustered regions, or repeatedly monitored sequential test.
estimated absolute lift ± 1.96 × estimated standard error
LAB 10
Confidence intervals · Northstar
CONTROL10.00%2,000 / 20,000 buyers
TREATMENT10.20%2,040 / 20,000 buyers
Absolute lift+0.20 pp
Relative lift2.00%
Two-sided p-value0.5069
Hold the observed lift fixed and increase the sample. The interval narrows and the null becomes less compatible with the estimate. This is a sensitivity demonstration, not a sequence of accumulating real outcomes. Interval: unpooled Wald normal approximation; p-value: pooled two-proportion z-test. Independent buyers, mature outcomes, one fixed-horizon analysis; not for sparse or clustered data.
Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER
The interval includes zero and a meaningful benefit. What can we conclude?