Layout
PART IV · Sharper estimates · CONCEPT 14 OF 24

A daily look is not a daily permission to stop.

EXPLORE~7 min
Back to journey
  • The team refreshes the dashboard every morning and wants to ship the first day p < 0.05.
  • Repeatedly applying a fixed-horizon rule changes its false-positive rate.
Peeking & stopping: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Observation and optional stopping differ.
  • Watching operational safety is sensible.
  • Repeatedly making efficacy decisions with ordinary fixed-horizon p-values is a different test from the one you planned.

Northstar can monitor outages for safety without treating every dashboard refresh as permission to claim a fixed-horizon conversion win.

Prespecify the analysis horizon.
  • For a fixed-horizon design, finish the sample, duration, and outcome maturation rules before making the final efficacy decision.
  • Do not extend only the disappointing tests.

Finish Northstar's prespecified sample, duration, and follow-up rules rather than extending only the cache tests that nearly look significant.

Sequential designs exist.
  • Group-sequential boundaries, alpha spending, and always-valid methods permit suitable monitoring when used correctly.
  • Do not retrofit a convenient sequential rule after seeing the path.
  • If early efficacy decisions are essential, Northstar chooses a valid sequential design before launch.
  • The independent-tests illustration is not its exact cumulative-peeking error rate.
LAB 14

Peeking & stopping · Northstar

At least one false alarm: 40.1%
One valid null test5%
Independent opportunities10
Any false alarm40.1%

This is the exact family-wise error calculation for independent 5% null tests: 1 − 0.95^k. Repeated looks at a cumulative experiment are correlated, so this is NOT their exact false-positive rate. It illustrates the extra-opportunities problem, not a sequential-testing boundary. Use fixed-horizon analysis or a properly designed sequential method.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

Can you repeatedly use ordinary 5% tests and stop at the first win while retaining 5% Type I error?

Your learning note

Private · just for you