Layout
PART IV · Sharper estimates · CONCEPT 16 OF 24

Use the past to remove noise, not to leak the future.

EXPLORE~8 min
Back to journey
  • Buyers who spent more before the test often spend more during it.
  • CUPED uses a pre-treatment covariate to adjust away predictable outcome variation while targeting the same mean treatment effect under appropriate assumptions.
CUPED: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Adjust, do not select.
  • A centered adjustment Yadj = Y − θ(X − mean X) can reduce noise when X predicts Y.
  • In a randomized test, pre-treatment X has equal expectation across arms, preserving the treatment contrast in expectation.

Northstar adjusts for predictable variation from pre-test spending without selecting only high-spending buyers or redefining the business outcome.

Correlation determines the ideal gain.
  • For one covariate and an optimal linear coefficient, ideal residual variance is Var(Y) × (1 − ρ²).
  • Correlation 0.6 gives 64% of the original variance, not 40%.
  • Real gains need estimation and validation.

In the ideal one-covariate calculation, correlation 0.6 leaves 64% of Northstar's raw variance and 80% of its raw standard error.

Keep covariates genuinely pre-treatment.
  • During-test latency or spend can be affected by treatment and is not a safe pre-treatment covariate.
  • Handle new buyers with no history explicitly and avoid fitting procedures that invalidate standard errors.
  • Last month's Northstar spend may be a useful covariate.
  • During-test latency can be changed by treatment and is not a safe pre-treatment covariate.
Yadj = Y − θ(X − mean X) ideal residual variance / original variance = 1 − ρ²
LAB 16

CUPED · Northstar

Raw100%
Adjusted64.0%
Ideal variance remaining64.0%
Standard error remaining80.0%
Equivalent sample multiplier1.56×

Theoretical single-covariate optimal linear adjustment: variance multiplier 1 − ρ². No coefficient is fitted here, and the sample multiplier is an idealized precision equivalence, not new buyers. Covariates must precede treatment; validate estimation, missing-history handling, and uncertainty on real data.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

Which is a suitable candidate covariate?

Your learning note

Private · just for you