Layout
PART VI · In practice · CONCEPT 24 OF 24

From an infrastructure idea to trustworthy evidence.

EXPLORE~8 min
Back to journey
  • Bring the journey together: Northstar is evaluating a new product-page cache.
  • The goal is not to ‘get a significant result’.
  • It is to learn whether the change is worth rolling out safely.
An experimentation workflow: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Design before exposure.
  • State the causal question and population.
  • Map shared resources.
  • Choose sticky assignment, a seven-day conversion outcome, latency diagnostics, guardrail margins, a planning effect, sample size, and a stopping rule.

Before Northstar changes routing, specify the causal question, shared resources, buyer population, conversion window, guardrails, sample, and stopping rule.

Validate and run deliberately.
  • Check A/A behavior and instrumentation.
  • Launch a monitored canary, ramp safely, audit SRM and missing outcomes, and retain everyone in the intended analysis population.
  • Do not stop opportunistically for efficacy.

Validate Northstar's instrumentation, canary safely, audit SRM and missing outcomes, and keep buyers in their intended assignment groups.

Analyze, decide, and remember.
  • Wait for mature outcomes.
  • Estimate effects and uncertainty with the correct unit and metric.
  • Apply the declared decision rule, document limitations, roll out with monitoring, and preserve what the experiment taught.

Estimate with the right unit and metric, apply Northstar's original decision rule, preserve a decision record, and monitor the actual rollout.

LAB 24

An experimentation workflow · Northstar

  1. Will assigning eligible logged-in buyers to the new cache improve seven-day purchase conversion? Distinguish buyer benefit from a cost-saving migration objective.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

The primary metric wins, but assignment logs are missing disproportionately in treatment. What next?

Your learning note

Private · just for you