PART I · Foundations · CONCEPT 2 OF 24
A faster service is a hypothesis, not a business result.
EXPLORE~6 min
Back to journey- The proposal: cache product details closer to buyers.
- The mechanism is lower waiting time; the hoped-for outcome is more completed purchases.
- Write both down before the first buyer enters the test.
| Concept | What it means | Northstar example |
|---|---|---|
| Make the chain explicit. |
|
|
| Choose a primary outcome and a window. |
| Count whether each eligible assigned Northstar buyer completes an order within seven days of first eligibility; wait for that window to mature. |
| Precommit to a decision. |
| Before launch, Northstar records conversion as primary, worthwhile effect size, guardrail margins, sample/duration, assignment unit, and stopping rule. |
LAB 02
Hypothesis & success · Northstar
CONTROL10.00%2,000 / 20,000 buyers
TREATMENT10.20%2,040 / 20,000 buyers
Absolute lift+0.20 pp
Relative lift2.00%
Two-sided p-value0.5069
20 basis points = 0.20 percentage points = 2% relative lift from a 10% baseline. Change the sample without changing the displayed rates: precision, not the unit conversion, changes. Interval: unpooled Wald normal approximation; p-value: pooled two-proportion z-test. Independent buyers, mature outcomes, one fixed-horizon analysis; not for sparse or clustered data.
Constructed teaching example · not Amazon data or a production experimentation engine
Which is the primary business outcome in our example?
Your learning note
Private · just for youFurther reading: Microsoft Experimentation Platform ↗NIST statistical handbook ↗