PART I · Foundations · CONCEPT 1 OF 24
Comparing people who chose something is not an experiment.
EXPLORE~5 min
Back to journey- Northstar is a fictional Amazon-like marketplace.
- Its infrastructure team builds a faster product-page service.
- Buyers who happen to reach the new service purchase more.
- Did speed cause that—or were they already different buyers?
| Concept | What it means | Northstar example |
|---|---|---|
| The counterfactual is missing. |
|
|
| Self-selection mixes cause and customer mix. |
| Sending mostly premium buyers to Northstar's new cache can create an apparent conversion win even if the cache changes nobody's purchase behavior. |
| Random assignment makes a fair starting point. |
|
|
LAB 01
Why randomize? · Northstar
Control · existing stack
20% premium
Expected conversion: 8.0%
Treatment · new cache
80% premium
Expected conversion: 17.0%
True within-type treatment effect0.00 pp
Apparent lift from buyer mix+9.00 pp
One buyer’s next 4 requestsA → A → A → A
Premium buyers have 20% conversion; other buyers have 5%, on either stack. Randomization shows equal mixes in expectation here, not guaranteed finite-sample balance. Dot colors show buyer types. Sticky routing keeps one buyer on one experience; repeated requests remain correlated.
Constructed teaching example · not Amazon data or a production experimentation engine
The faster stack has mostly premium buyers. What is the clean next step?
Your learning note
Private · just for youFurther reading: Microsoft Experimentation Platform ↗NIST statistical handbook ↗