Layout
PART II · Measurement · CONCEPT 8 OF 24

Before testing the idea, test the experiment.

EXPLORE~7 min
Back to journey
  • An A/A sends both groups through the same implementation.
  • It helps check bucketing, logs, and analysis.
  • In an A/B, unexpected group counts can reveal a sample ratio mismatch (SRM).
A/A, logging & SRM: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Compare with the intended allocation.
  • A 50/50 test need not have exactly equal counts.
  • Compare observed independent assignment counts with the configured expected split using a suitable goodness-of-fit test.

For Northstar's configured 50/50 assignment, compare independent buyer counts with that expected split instead of requiring exact equality.

A mismatch is a diagnostic alarm.
  • Broken routing, bot filtering, identity joins, treatment-dependent missing logs, or eligibility differences can cause SRM.
  • Do not repair it by arbitrarily dropping extra buyers.

A large allocation mismatch prompts Northstar to inspect routing, identity joins, bot filters, and differential missing logs before interpreting conversion.

No alarm is not a certificate.
  • An A/A can still produce a false positive.
  • No detected SRM does not prove correct measurement.
  • Reconcile assignment, exposure, and outcome logs, including failures in both groups.

An A/A with no detected Northstar mismatch does not certify the entire pipeline; reconcile assignment, exposure, and failed-outcome records too.

LAB 08

A/A, logging & SRM · Northstar

A · 4,900
B · 5,100
Expected allocation50 / 50
SRM p-value (approx.)0.04550
Example alarm at p < 0.001No alarm

With 10,000 independent assignments, small differences happen by chance. This two-sided normal approximation is equivalent to a one-degree-of-freedom chi-square large-sample test. The alarm threshold is illustrative, not a universal rule. No alarm does not certify the pipeline.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

A severe SRM alarm appears. What next?

Your learning note

Private · just for you