PART VI · In practice · CONCEPT 21 OF 24
A first-day win may not survive a full week.
EXPLORE~7 min
Back to journey- Mobile buyers on slow networks may benefit most.
- Cache warmup can also make day one unlike day seven.
- Northstar must separate genuine heterogeneity from noisy slices and changing conditions.
| Concept | What it means | Northstar example |
|---|---|---|
| Segment on pre-treatment attributes. |
| Segment Northstar buyers by device or region recorded before exposure, not by a purchase pattern treatment could have caused during the test. |
| Test differences, not significance labels. |
| Compare the treatment effects between Northstar's mobile and desktop groups with an interaction analysis, not their separate significance labels. |
| Allow the system and outcomes to settle. |
| Northstar predefines warmup, weekday coverage, and outcome maturation instead of dropping disappointing early cache days after seeing them. |
LAB 21
Segments & time · Northstar
Constructed daily absolute lift (pp)
Supplied mobile effect+0.60 pp
Supplied desktop effect+0.10 pp
Steady-state weighted effect+0.35 pp
True segment effects and a warmup curve are supplied, not estimated. Changing population mix changes the aggregate even if segment effects stay fixed. A real interaction needs uncertainty and multiplicity-aware testing; this smooth illustration contains no evidence that early days may be discarded.
Constructed teaching example · not Amazon data or a production experimentation engine
Mobile is significant; desktop is not. Does that establish an interaction?
Your learning note
Private · just for youFurther reading: Microsoft Experimentation Platform ↗NIST statistical handbook ↗