PART I · Foundations · CONCEPT 4 OF 24
Count everyone assigned, including the unlucky buyers.
EXPLORE~6 min
Back to journey- Some buyers assigned to the new cache hit a fallback path.
- Others see a failed page and leave.
- Excluding those buyers can turn reliability failures into an apparent conversion improvement.
| Concept | What it means | Northstar example |
|---|---|---|
| Assignment defines the comparison. |
| A Northstar buyer assigned to the new stack stays in treatment even if the system falls back to the old path. |
| Do not select on treatment outcomes. |
| Dropping Northstar buyers whose treatment page failed removes a potential treatment-caused harm from the conversion denominator. |
| Exposure analysis needs a defensible trigger. |
| Northstar can investigate a trigger logged symmetrically in both groups, not define treatment exposure as successfully using the new cache. |
LAB 04
Intent to treat · Northstar
Local absolute effect (supplied)+0.80 pp
Expected all-buyer effect+0.20 pp
Separate selection example10.20%
Selection-example denominator1,000 assigned buyers
Left: dilution assumes a fixed, treatment-independent affected share and zero effect elsewhere. Right: a separate constructed treatment group has 102 purchasers among 1,000 assigned buyers. Dropping 100 failed-load non-purchasers falsely raises reported conversion to 102/900. These are two distinct illustrations, not one fitted model.
Constructed teaching example · not Amazon data or a production experimentation engine
Should failed new-stack loads be removed from treatment conversion?
Your learning note
Private · just for youFurther reading: Microsoft Experimentation Platform ↗NIST statistical handbook ↗