Layout
PART V · Hidden traps · CONCEPT 19 OF 24

Enough scorecards will eventually find a winner.

EXPLORE~6 min
Back to journey
  • The dashboard has conversion, revenue, latency, 20 countries, and 10 device groups.
  • Searching them all for any p < 0.05 creates more opportunities for a false discovery.
Multiple comparisons: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Define the family of claims.
  • One predeclared primary outcome prevents a post-hoc metric beauty contest.
  • Multiple variants, co-primary metrics, and confirmatory subgroup claims may still need multiplicity control.

Northstar declares the primary metric and confirmatory segment claims before searching scores across countries, devices, and outcomes.

Match the correction to the objective.
  • Holm or Bonferroni can control family-wise error.
  • False discovery rate procedures target a different error criterion.
  • Neither makes a statistically tiny effect economically valuable.

Northstar chooses family-wise error or false-discovery control according to its claims; one correction is not interchangeable with every other objective.

Label exploration honestly.
  • A surprising device segment can generate a hypothesis.
  • It is not automatically a confirmed win.
  • Replicate or test the interaction with an appropriate analysis and correction.
  • A surprising Northstar mobile slice can motivate another test.
  • A significant mobile result and insignificant desktop result alone do not prove an interaction.
LAB 19

Multiple comparisons · Northstar

Unadjusted any-false-alarm risk64.2%
Bonferroni per-test alpha0.00250
Bonferroni family-wise bound≤ 5%

Unadjusted risk assumes independent true-null tests; correlated dashboard metrics have a different joint rate. The Bonferroni bound applies regardless of dependence when individual tests are valid. This illustrates family-wise error, not false discovery rate or sequential monitoring.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

Does one significant segment prove the treatment works differently there?

Your learning note

Private · just for you