Layout
PART II · Measurement · CONCEPT 5 OF 24

A tenth of a point is not a tenth of a percent.

EXPLORE~5 min
Back to journey
  • Suppose control converts at 10.0% and treatment at 10.2%.
  • That is a 0.2 percentage-point increase, or a 2% relative lift.
  • Both are correct; they answer different questions.
Conversion & lift: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Count buyers, not orders.
  • Seven-day buyer conversion is buyers with at least one order divided by all eligible assigned buyers with complete follow-up.
  • Multiple orders from one buyer still contribute one conversion.

A Northstar buyer with three orders contributes one converted buyer, not three conversions, to the seven-day buyer-conversion rate.

Keep units visible.
  • Absolute lift is treatment rate minus control rate.
  • Relative lift divides that difference by the control rate.
  • A 2% relative lift from a 10% baseline is 0.2 percentage points, not 2 points.
  • Northstar's 10.0% → 10.2% example is +0.2 percentage points and +2% relative lift.
  • The same change has two different units.
Point estimates need uncertainty.
  • The same observed lift can be compelling with many independent buyers and inconclusive with few.
  • Do not extrapolate revenue from a noisy estimate without acknowledging its range.

An observed +0.2-point Northstar lift is much less precise with 1,000 buyers per arm than with hundreds of thousands.

absolute lift = pT − pC relative lift = (pT − pC) / pC
LAB 05

Conversion & lift · Northstar

CONTROL10.00%2,000 / 20,000 buyers
TREATMENT10.20%2,040 / 20,000 buyers
zero effect · -0.39 pp to +0.79 pp
Absolute lift+0.20 pp
Relative lift2.00%
Two-sided p-value0.5069

20 basis points = 0.20 percentage points = 2% relative lift from a 10% baseline. Change the sample without changing the displayed rates: precision, not the unit conversion, changes. Interval: unpooled Wald normal approximation; p-value: pooled two-proportion z-test. Independent buyers, mature outcomes, one fixed-horizon analysis; not for sparse or clustered data.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

10.0% → 10.2% is what relative lift?

Your learning note

Private · just for you