PART II · Measurement · CONCEPT 7 OF 24
A green speed chart cannot excuse broken checkout.
EXPLORE~6 min
Back to journey- Northstar wants faster pages, but not at the cost of stale prices, checkout failures, or an exploding infrastructure bill.
- Guardrails protect outcomes that the primary metric can miss.
| Concept | What it means | Northstar example |
|---|---|---|
| Declare harm margins. |
| Before testing the cache, Northstar declares an acceptable checkout-error increase, with immediate rollback rules for severe correctness incidents. |
| Not significant is not safe. |
|
|
| Separate benefits, costs, and safety. |
| Read conversion as buyer outcome, latency as mechanism, errors/stale prices as safety, and cost per buyer as economics—not a post-hoc composite win score. |
LAB 07
Guardrails & trade-offs · Northstar
ERROR GUARDRAIL · LOWER IS BETTER
Unacceptable harm is not ruled out
One-sided 95% upper bound: 21.4 bp. Predeclared acceptable increase: 10 bp (0.10 pp).
Observed increase5 bp
Upper confidence bound21.4 bp
Prespecified margin10 bp
Reduce the standard error while keeping the observed error increase fixed. A small, uncertain harm estimate is not evidence of safety. This is a normal-approximation bound with supplied estimates, for one predeclared guardrail; multiple guardrails and critical incidents require an appropriate policy.
Constructed teaching example · not Amazon data or a production experimentation engine
An error increase is not significant but its interval includes serious harm. Is safety established?
Your learning note
Private · just for youFurther reading: Microsoft Experimentation Platform ↗NIST statistical handbook ↗