PART V · Hidden traps · CONCEPT 19 OF 24
Enough scorecards will eventually find a winner.
EXPLORE~6 min
Back to journey- The dashboard has conversion, revenue, latency, 20 countries, and 10 device groups.
- Searching them all for any p < 0.05 creates more opportunities for a false discovery.
| Concept | What it means | Northstar example |
|---|---|---|
| Define the family of claims. |
| Northstar declares the primary metric and confirmatory segment claims before searching scores across countries, devices, and outcomes. |
| Match the correction to the objective. |
| Northstar chooses family-wise error or false-discovery control according to its claims; one correction is not interchangeable with every other objective. |
| Label exploration honestly. |
|
|
LAB 19
Multiple comparisons · Northstar
Unadjusted any-false-alarm risk64.2%
Bonferroni per-test alpha0.00250
Bonferroni family-wise bound≤ 5%
Unadjusted risk assumes independent true-null tests; correlated dashboard metrics have a different joint rate. The Bonferroni bound applies regardless of dependence when individual tests are valid. This illustrates family-wise error, not false discovery rate or sequential monitoring.
Constructed teaching example · not Amazon data or a production experimentation engine
Does one significant segment prove the treatment works differently there?
Your learning note
Private · just for youFurther reading: Microsoft Experimentation Platform ↗NIST statistical handbook ↗