PART IV · Sharper estimates · CONCEPT 14 OF 24
A daily look is not a daily permission to stop.
EXPLORE~7 min
Back to journey- The team refreshes the dashboard every morning and wants to ship the first day p < 0.05.
- Repeatedly applying a fixed-horizon rule changes its false-positive rate.
| Concept | What it means | Northstar example |
|---|---|---|
| Observation and optional stopping differ. |
| Northstar can monitor outages for safety without treating every dashboard refresh as permission to claim a fixed-horizon conversion win. |
| Prespecify the analysis horizon. |
| Finish Northstar's prespecified sample, duration, and follow-up rules rather than extending only the cache tests that nearly look significant. |
| Sequential designs exist. |
|
|
LAB 14
Peeking & stopping · Northstar
At least one false alarm: 40.1%
One valid null test5%
Independent opportunities10
Any false alarm40.1%
This is the exact family-wise error calculation for independent 5% null tests: 1 − 0.95^k. Repeated looks at a cumulative experiment are correlated, so this is NOT their exact false-positive rate. It illustrates the extra-opportunities problem, not a sequential-testing boundary. Use fixed-horizon analysis or a properly designed sequential method.
Constructed teaching example · not Amazon data or a production experimentation engine
Can you repeatedly use ordinary 5% tests and stop at the first win while retaining 5% Type I error?
Your learning note
Private · just for youFurther reading: Microsoft Experimentation Platform ↗NIST statistical handbook ↗