From an infrastructure idea to trustworthy evidence.
- Bring the journey together: Northstar is evaluating a new product-page cache.
- The goal is not to ‘get a significant result’.
- It is to learn whether the change is worth rolling out safely.
| Concept | What it means | Northstar example |
|---|---|---|
| Design before exposure. |
| Before Northstar changes routing, specify the causal question, shared resources, buyer population, conversion window, guardrails, sample, and stopping rule. |
| Validate and run deliberately. |
| Validate Northstar's instrumentation, canary safely, audit SRM and missing outcomes, and keep buyers in their intended assignment groups. |
| Analyze, decide, and remember. |
| Estimate with the right unit and metric, apply Northstar's original decision rule, preserve a decision record, and monitor the actual rollout. |
An experimentation workflow · Northstar
Will assigning eligible logged-in buyers to the new cache improve seven-day purchase conversion? Distinguish buyer benefit from a cost-saving migration objective.
Identify shared queues and cache state. Choose sticky buyer buckets only if the contrast is meaningful; otherwise design isolated pools, clusters, or randomized switchbacks.
Primary: seven-day conversion. Mechanism: page latency and tail. Guardrails: errors, stale prices, checkout failures. Declare margins, metric definitions, multiplicity, and cost evaluation.
Use distinct buyers and a worthwhile absolute effect. Plan power, business cycles, seven-day follow-up, and a fixed horizon or valid sequential procedure.
Reconcile assignment and outcomes, inspect A/A calibration, verify rollback, and launch a small safety canary. Investigate SRM and differential missingness.
Ramp safely, track configuration and phase boundaries, preserve original assignment, monitor safety, and wait for outcome windows. Do not cherry-pick dates or stop at a lucky p-value.
Use the right unit and metric-specific inference, report effect intervals, verify guardrail bounds, and apply the original objective. Label exploratory segments and unresolved interference.
Validate full-load behavior, expand in stages, monitor guardrails, preserve a decision record, and revisit the result if the system or buyer population changes.