Layout
PART I · Foundations · CONCEPT 3 OF 24

A buyer should not switch teams on every click.

EXPLORE~6 min
Back to journey
  • Northstar buyers browse repeatedly and may use several devices.
  • Randomizing each request creates mixed experiences and correlated observations.
  • Our default is a stable buyer-ID assignment for logged-in eligible buyers.
Unit of randomization: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Keep assignment sticky.
  • Hash a stable buyer ID with an experiment-specific salt into a bucket.
  • Route that buyer consistently throughout the test.
  • Define anonymous users and identity changes separately rather than silently reassigning them.

A stable Northstar buyer ID stays on the same cache variant across repeat visits instead of switching treatment on every request.

Analyze at the unit you randomized.
  • Ten requests from one buyer are not ten independent buyers.
  • Aggregate outcomes per buyer or use inference that respects buyer clusters.
  • Request-level telemetry is useful but cannot manufacture extra independent sample size.
  • One buyer's 100 Northstar requests are still one randomized buyer.
  • Aggregate appropriately or use buyer-cluster-aware inference.
Shared infrastructure can force another design.
  • If cache, queues, sellers, or regional load connect the groups, buyer randomization may not isolate the change.
  • Consider isolated pools, cluster assignment, or time-block switchbacks with appropriate inference.

If the new cache changes a database queue shared with control, Northstar may need isolated pools, clusters, or randomized time blocks rather than ordinary buyer buckets.

LAB 03

Unit of randomization · Northstar

Control · existing stack
20% premium

Expected conversion: 8.0%

Treatment · new cache
80% premium

Expected conversion: 17.0%

True within-type treatment effect0.00 pp
Apparent lift from buyer mix+9.00 pp
One buyer’s next 4 requestsA → A → A → A

Premium buyers have 20% conversion; other buyers have 5%, on either stack. Randomization shows equal mixes in expectation here, not guaranteed finite-sample balance. Dot colors show buyer types. Sticky routing keeps one buyer on one experience; repeated requests remain correlated.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

A buyer makes 100 requests. How many independent randomized buyers is that?

Your learning note

Private · just for you