PART II · Measurement · CONCEPT 5 OF 24
A tenth of a point is not a tenth of a percent.
EXPLORE~5 min
Back to journey- Suppose control converts at 10.0% and treatment at 10.2%.
- That is a 0.2 percentage-point increase, or a 2% relative lift.
- Both are correct; they answer different questions.
| Concept | What it means | Northstar example |
|---|---|---|
| Count buyers, not orders. |
| A Northstar buyer with three orders contributes one converted buyer, not three conversions, to the seven-day buyer-conversion rate. |
| Keep units visible. |
|
|
| Point estimates need uncertainty. |
| An observed +0.2-point Northstar lift is much less precise with 1,000 buyers per arm than with hundreds of thousands. |
absolute lift = pT − pC
relative lift = (pT − pC) / pC
LAB 05
Conversion & lift · Northstar
CONTROL10.00%2,000 / 20,000 buyers
TREATMENT10.20%2,040 / 20,000 buyers
Absolute lift+0.20 pp
Relative lift2.00%
Two-sided p-value0.5069
20 basis points = 0.20 percentage points = 2% relative lift from a 10% baseline. Change the sample without changing the displayed rates: precision, not the unit conversion, changes. Interval: unpooled Wald normal approximation; p-value: pooled two-proportion z-test. Independent buyers, mature outcomes, one fixed-horizon analysis; not for sparse or clustered data.
Constructed teaching example · not Amazon data or a production experimentation engine
10.0% → 10.2% is what relative lift?
Your learning note
Private · just for youFurther reading: Microsoft Experimentation Platform ↗NIST statistical handbook ↗