Layout
PART I · Foundations · CONCEPT 1 OF 24

Comparing people who chose something is not an experiment.

EXPLORE~5 min
Back to journey
  • Northstar is a fictional Amazon-like marketplace.
  • Its infrastructure team builds a faster product-page service.
  • Buyers who happen to reach the new service purchase more.
  • Did speed cause that—or were they already different buyers?
Why randomize?: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
The counterfactual is missing.
  • We cannot watch the same buyer use both versions at the same moment.
  • We need a comparable group to stand in for what would have happened without the change.
  • Northstar cannot observe the same buyer using old and new infrastructure at the same moment.
  • Randomly assigned control buyers stand in for that missing alternative.
Self-selection mixes cause and customer mix.
  • Sending premium buyers or newer devices to the new stack can make it appear better even if it has no effect.
  • A before/after comparison also mixes the change with campaigns, seasonality, and outages.

Sending mostly premium buyers to Northstar's new cache can create an apparent conversion win even if the cache changes nobody's purchase behavior.

Random assignment makes a fair starting point.
  • Assign eligible buyers to old or new infrastructure by chance.
  • In expectation, the groups have the same mix.
  • Real finite samples can still be imbalanced, and broken instrumentation can still ruin the test.
  • Assign eligible Northstar buyers by chance before exposure, then audit routing and logs.
  • Randomization balances the mix in expectation, not perfectly in every sample.
LAB 01

Why randomize? · Northstar

Control · existing stack
20% premium

Expected conversion: 8.0%

Treatment · new cache
80% premium

Expected conversion: 17.0%

True within-type treatment effect0.00 pp
Apparent lift from buyer mix+9.00 pp
One buyer’s next 4 requestsA → A → A → A

Premium buyers have 20% conversion; other buyers have 5%, on either stack. Randomization shows equal mixes in expectation here, not guaranteed finite-sample balance. Dot colors show buyer types. Sticky routing keeps one buyer on one experience; repeated requests remain correlated.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

The faster stack has mostly premium buyers. What is the clean next step?

Your learning note

Private · just for you