Layout
PART VI · In practice · CONCEPT 21 OF 24

A first-day win may not survive a full week.

EXPLORE~7 min
Back to journey
  • Mobile buyers on slow networks may benefit most.
  • Cache warmup can also make day one unlike day seven.
  • Northstar must separate genuine heterogeneity from noisy slices and changing conditions.
Segments & time: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Segment on pre-treatment attributes.
  • Device class or region defined before exposure is safer than ‘buyers who purchased twice during the test’.
  • Prespecify important segments and distinguish exploratory views from confirmatory claims.

Segment Northstar buyers by device or region recorded before exposure, not by a purchase pattern treatment could have caused during the test.

Test differences, not significance labels.
  • A significant mobile result and an insignificant desktop result do not prove different effects.
  • Examine a treatment-by-segment interaction and its uncertainty, with appropriate multiplicity handling.

Compare the treatment effects between Northstar's mobile and desktop groups with an interaction analysis, not their separate significance labels.

Allow the system and outcomes to settle.
  • Warmup, novelty, day-of-week mix, and delayed conversions affect interpretation.
  • Predefine windows and maturation; do not delete inconvenient early days after seeing them.

Northstar predefines warmup, weekday coverage, and outcome maturation instead of dropping disappointing early cache days after seeing them.

LAB 21

Segments & time · Northstar

Constructed daily conversion lift in percentage points0.10.20.30.4Day 1Day 7Day 14
Constructed daily absolute lift (pp)
Supplied mobile effect+0.60 pp
Supplied desktop effect+0.10 pp
Steady-state weighted effect+0.35 pp

True segment effects and a warmup curve are supplied, not estimated. Changing population mix changes the aggregate even if segment effects stay fixed. A real interaction needs uncertainty and multiplicity-aware testing; this smooth illustration contains no evidence that early days may be discarded.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

Mobile is significant; desktop is not. Does that establish an interaction?

Your learning note

Private · just for you