Layout
PART VI · In practice · CONCEPT 23 OF 24

Make a decision, not just a significance announcement.

EXPLORE~8 min
Back to journey
  • The experiment ends.
  • The team needs a decision that combines buyer impact, practical value, reliability, costs, and remaining uncertainty—not just one green number.
Ship, hold, or stop: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Read the whole scorecard.
  • First establish trustworthy assignment and mature outcomes.
  • Then inspect the primary effect and interval, mechanism metrics, guardrail bounds, costs, and known limitations.

Northstar first checks trustworthy assignment and mature outcomes, then reads conversion intervals, latency, guardrail bounds, cost, and limitations together.

Separate superiority from non-inferiority.
  • A cost-saving migration may be worthwhile if buyer impact is demonstrably above a predeclared acceptable-loss margin.
  • That is a different decision rule from proving conversion uplift; do not switch objectives after a null result.

A cost-saving Northstar migration with conversion non-inferiority is a different predeclared objective from proving a conversion uplift.

Hold is an honest outcome.
  • If the interval still allows important harm, a safety bound fails, or interference makes the target unclear, hold or redesign.
  • More data helps sampling uncertainty, not broken logging or an inappropriate design.

If Northstar's error bound permits unacceptable harm or logs are biased, hold or redesign; extra buyers do not repair broken measurement.

LAB 23

Ship, hold, or stop · Northstar

Supplied conversion interval−0.15 to +0.55 pp
Diagnostic / guardrail statusHarm bound exceeds margin
DISCUSS THE DECISION

Hold: both buyer impact and safety need resolution. More buyers help precision, not a broken design.

Hypothetical completed scorecards, not fitted experiments. A real decision requires the originally specified tests, bounds, operational evidence, and business objective. No universal ‘ship if green’ rule is implemented.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

Conversion is inconclusive, but costs fall. Can you invent a non-inferiority margin now?

Your learning note

Private · just for you