Layout
PART II · Measurement · CONCEPT 7 OF 24

A green speed chart cannot excuse broken checkout.

EXPLORE~6 min
Back to journey
  • Northstar wants faster pages, but not at the cost of stale prices, checkout failures, or an exploding infrastructure bill.
  • Guardrails protect outcomes that the primary metric can miss.
Guardrails & trade-offs: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Declare harm margins.
  • Set acceptable changes before launch: for example, checkout failure may rise by no more than 0.1 percentage points.
  • Safety-critical incidents can require immediate shutdown regardless of statistical significance.

Before testing the cache, Northstar declares an acceptable checkout-error increase, with immediate rollback rules for severe correctness incidents.

Not significant is not safe.
  • A wide interval spanning substantial harm and benefit is inconclusive.
  • Demonstrating non-inferiority requires a suitable one-sided test or confidence bound against a declared harm margin.
  • An insignificant Northstar error increase with a wide interval still permits harm.
  • Compare the appropriate upper bound with the declared margin.
Separate benefits, costs, and safety.
  • Conversion is our primary outcome; latency explains the mechanism; errors and stale data are guardrails; cost per buyer informs the trade-off.
  • Do not combine them into a convenient post-hoc score.

Read conversion as buyer outcome, latency as mechanism, errors/stale prices as safety, and cost per buyer as economics—not a post-hoc composite win score.

LAB 07

Guardrails & trade-offs · Northstar

ERROR GUARDRAIL · LOWER IS BETTER

Unacceptable harm is not ruled out

One-sided 95% upper bound: 21.4 bp. Predeclared acceptable increase: 10 bp (0.10 pp).

Observed increase5 bp
Upper confidence bound21.4 bp
Prespecified margin10 bp

Reduce the standard error while keeping the observed error increase fixed. A small, uncertain harm estimate is not evidence of safety. This is a normal-approximation bound with supplied estimates, for one predeclared guardrail; multiple guardrails and critical incidents require an appropriate policy.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

An error increase is not significant but its interval includes serious harm. Is safety established?

Your learning note

Private · just for you