Layout
PART III · Inference · CONCEPT 10 OF 24

The range matters as much as the dot.

EXPLORE~7 min
Back to journey
  • Our estimated conversion lift is 0.2 points.
  • An interval from −0.6 to +1.0 points supports a very different decision from one spanning +0.1 to +0.3 points.
Confidence intervals: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Interpret the procedure.
  • A 95% confidence procedure covers the fixed true effect in about 95% of repeated experiments under its assumptions.
  • It does not assign a 95% probability to this realized interval containing that fixed effect.

Across repeated appropriate experiments, Northstar's 95% interval procedure should cover the fixed true effect about 95% of the time.

Compare with zero and with worthwhile effects.
  • An interval can exclude zero but include only commercially tiny effects.
  • Another can include zero while still permitting an important gain or loss.
  • ‘Not significant’ does not mean ‘no effect’.

Northstar distinguishes a tiny, precise conversion improvement from a wide interval that still permits both meaningful benefit and harm.

Check the approximation.
  • The playground uses a large-sample normal interval for a difference of independent buyer conversion proportions.
  • Sparse events, clusters, sequential looks, and selection require different treatment.

The normal-proportion illustration is not automatically suitable for Northstar's sparse errors, clustered regions, or repeatedly monitored sequential test.

estimated absolute lift ± 1.96 × estimated standard error
LAB 10

Confidence intervals · Northstar

CONTROL10.00%2,000 / 20,000 buyers
TREATMENT10.20%2,040 / 20,000 buyers
zero effect · -0.39 pp to +0.79 pp
Absolute lift+0.20 pp
Relative lift2.00%
Two-sided p-value0.5069

Hold the observed lift fixed and increase the sample. The interval narrows and the null becomes less compatible with the estimate. This is a sensitivity demonstration, not a sequence of accumulating real outcomes. Interval: unpooled Wald normal approximation; p-value: pooled two-proportion z-test. Independent buyers, mature outcomes, one fixed-horizon analysis; not for sparse or clustered data.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

The interval includes zero and a meaningful benefit. What can we conclude?

Your learning note

Private · just for you