The team needs a decision that combines buyer impact, practical value, reliability, costs, and remaining uncertainty—not just one green number.
Ship, hold, or stop: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
Concept
What it means
Northstar example
📊Read the whole scorecard.
First establish trustworthy assignment and mature outcomes.
Then inspect the primary effect and interval, mechanism metrics, guardrail bounds, costs, and known limitations.
🛒 Northstar first checks trustworthy assignment and mature outcomes, then reads conversion intervals, latency, guardrail bounds, cost, and limitations together.
⚖️Separate superiority from non-inferiority.
A cost-saving migration may be worthwhile if buyer impact is demonstrably above a predeclared acceptable-loss margin.
That is a different decision rule from proving conversion uplift; do not switch objectives after a null result.
🛒 A cost-saving Northstar migration with conversion non-inferiority is a different predeclared objective from proving a conversion uplift.
⏸️Hold is an honest outcome.
If the interval still allows important harm, a safety bound fails, or interference makes the target unclear, hold or redesign.
More data helps sampling uncertainty, not broken logging or an inappropriate design.
🛒 If Northstar's error bound permits unacceptable harm or logs are biased, hold or redesign; extra buyers do not repair broken measurement.
Hold: both buyer impact and safety need resolution. More buyers help precision, not a broken design.
Hypothetical completed scorecards, not fitted experiments. A real decision requires the originally specified tests, bounds, operational evidence, and business objective. No universal ‘ship if green’ rule is implemented.
Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER
Conversion is inconclusive, but costs fall. Can you invent a non-inferiority margin now?