PART II · Measurement · CONCEPT 8 OF 24
Before testing the idea, test the experiment.
EXPLORE~7 min
Back to journey- An A/A sends both groups through the same implementation.
- It helps check bucketing, logs, and analysis.
- In an A/B, unexpected group counts can reveal a sample ratio mismatch (SRM).
| Concept | What it means | Northstar example |
|---|---|---|
| Compare with the intended allocation. |
| For Northstar's configured 50/50 assignment, compare independent buyer counts with that expected split instead of requiring exact equality. |
| A mismatch is a diagnostic alarm. |
| A large allocation mismatch prompts Northstar to inspect routing, identity joins, bot filters, and differential missing logs before interpreting conversion. |
| No alarm is not a certificate. |
| An A/A with no detected Northstar mismatch does not certify the entire pipeline; reconcile assignment, exposure, and failed-outcome records too. |
LAB 08
A/A, logging & SRM · Northstar
Expected allocation50 / 50
SRM p-value (approx.)0.04550
Example alarm at p < 0.001No alarm
With 10,000 independent assignments, small differences happen by chance. This two-sided normal approximation is equivalent to a one-degree-of-freedom chi-square large-sample test. The alarm threshold is illustrative, not a universal rule. No alarm does not certify the pipeline.
Constructed teaching example · not Amazon data or a production experimentation engine
A severe SRM alarm appears. What next?
Your learning note
Private · just for youFurther reading: Microsoft Experimentation Platform ↗NIST statistical handbook ↗