Layout
PART I · Foundations · CONCEPT 2 OF 24

A faster service is a hypothesis, not a business result.

EXPLORE~6 min
Back to journey
  • The proposal: cache product details closer to buyers.
  • The mechanism is lower waiting time; the hoped-for outcome is more completed purchases.
  • Write both down before the first buyer enters the test.
Hypothesis & success: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Make the chain explicit.
  • New cache → lower product-page latency → less abandonment → higher purchase conversion.
  • Each arrow is an assumption.
  • Latency can improve without a detectable conversion lift.
  • Northstar's proposed chain is cache change → faster product pages → less abandonment → more purchasers.
  • A latency improvement alone does not confirm every link.
Choose a primary outcome and a window.
  • For this journey, conversion is at least one completed order within seven days of a buyer’s first eligible visit.
  • Define eligibility before assignment and wait for every included buyer’s window to mature.

Count whether each eligible assigned Northstar buyer completes an order within seven days of first eligibility; wait for that window to mature.

Precommit to a decision.
  • Record one primary metric, the smallest worthwhile effect, duration, assignment unit, analysis method, and guardrail margins.
  • Engineering-only improvements can justify a different predeclared objective, such as cost savings with conversion non-inferiority.

Before launch, Northstar records conversion as primary, worthwhile effect size, guardrail margins, sample/duration, assignment unit, and stopping rule.

LAB 02

Hypothesis & success · Northstar

CONTROL10.00%2,000 / 20,000 buyers
TREATMENT10.20%2,040 / 20,000 buyers
zero effect · -0.39 pp to +0.79 pp
Absolute lift+0.20 pp
Relative lift2.00%
Two-sided p-value0.5069

20 basis points = 0.20 percentage points = 2% relative lift from a 10% baseline. Change the sample without changing the displayed rates: precision, not the unit conversion, changes. Interval: unpooled Wald normal approximation; p-value: pooled two-proportion z-test. Independent buyers, mature outcomes, one fixed-horizon analysis; not for sparse or clustered data.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

Which is the primary business outcome in our example?

Your learning note

Private · just for you