PART IV · Sharper estimates · CONCEPT 16 OF 24
Use the past to remove noise, not to leak the future.
EXPLORE~8 min
Back to journey- Buyers who spent more before the test often spend more during it.
- CUPED uses a pre-treatment covariate to adjust away predictable outcome variation while targeting the same mean treatment effect under appropriate assumptions.
| Concept | What it means | Northstar example |
|---|---|---|
| Adjust, do not select. |
| Northstar adjusts for predictable variation from pre-test spending without selecting only high-spending buyers or redefining the business outcome. |
| Correlation determines the ideal gain. |
| In the ideal one-covariate calculation, correlation 0.6 leaves 64% of Northstar's raw variance and 80% of its raw standard error. |
| Keep covariates genuinely pre-treatment. |
|
|
Yadj = Y − θ(X − mean X)
ideal residual variance / original variance = 1 − ρ²
LAB 16
CUPED · Northstar
Ideal variance remaining64.0%
Standard error remaining80.0%
Equivalent sample multiplier1.56×
Theoretical single-covariate optimal linear adjustment: variance multiplier 1 − ρ². No coefficient is fitted here, and the sample multiplier is an idealized precision equivalence, not new buyers. Covariates must precede treatment; validate estimation, missing-history handling, and uncertainty on real data.
Constructed teaching example · not Amazon data or a production experimentation engine
Which is a suitable candidate covariate?
Your learning note
Private · just for youFurther reading: Microsoft Experimentation Platform ↗NIST statistical handbook ↗