Layout
PART VI · In practice · CONCEPT 22 OF 24

A 1% test is not a 100% load test.

EXPLORE~7 min
Back to journey
  • A tiny canary checks basic safety, but a cache that works at 1% can fail at full traffic.
  • Warmup, contention, and capacity can make treatment effects depend on saturation.
Safe ramps & load: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Ramp to protect buyers.
  • Start with a small safety canary, monitor correctness and severe incidents, then increase under predeclared checks.
  • Rollback should be operationally tested before exposure.

Northstar starts with a monitored safety canary and a tested rollback path before exposing a large share of buyers.

Keep cohorts and phases interpretable.
  • Changing allocation, configuration, and system state can change the target effect.
  • Log phase boundaries and use an analysis appropriate for the ramp rather than pooling every phase blindly.

Log Northstar's allocation, configuration, and capacity phase boundaries; pooling a 1% canary and a saturated full ramp blindly can change the question.

Combine different kinds of evidence.
  • An A/B informs buyer impact at the tested saturation.
  • Load tests, fault injection, capacity models, and staged rollouts address operational limits.
  • None is a substitute for the others.
  • A buyer A/B estimates impact at tested traffic.
  • Northstar also needs load tests and capacity evidence before assuming the same behavior at full rollout.
LAB 22

Safe ramps & load · Northstar

SATURATION SCHEMATIC

Below the supplied threshold

Constructed error rate: 0.10%. Small-scale success does not predict the whole curve.

This deliberately simple nonlinear load illustration is not a capacity forecast. Ramps require measured saturation, rollback capability, phase-aware analysis, and safety monitoring. Do not interpret the supplied threshold as an engineering recommendation.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

A canary succeeds at 1% traffic. What is still unknown?

Your learning note

Private · just for you