Layout
PART I · Foundations · CONCEPT 4 OF 24

Count everyone assigned, including the unlucky buyers.

EXPLORE~6 min
Back to journey
  • Some buyers assigned to the new cache hit a fallback path.
  • Others see a failed page and leave.
  • Excluding those buyers can turn reliability failures into an apparent conversion improvement.
Intent to treat: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Assignment defines the comparison.
  • Intent-to-treat compares all eligible buyers by original assignment, even if delivery is imperfect.
  • It estimates the effect of assigning the new system under the tested operating conditions.

A Northstar buyer assigned to the new stack stays in treatment even if the system falls back to the old path.

Do not select on treatment outcomes.
  • Filtering to buyers whose new page loaded successfully removes failures caused by the change.
  • Filtering to purchasers is worse: it selects directly on the outcome.

Dropping Northstar buyers whose treatment page failed removes a potential treatment-caused harm from the conversion denominator.

Exposure analysis needs a defensible trigger.
  • A pre-treatment eligibility condition can sharpen the question.
  • A post-assignment trigger needs a counterfactual equivalent logged in both groups and an audit for treatment-dependent selection.

Northstar can investigate a trigger logged symmetrically in both groups, not define treatment exposure as successfully using the new cache.

LAB 04

Intent to treat · Northstar

Local absolute effect (supplied)+0.80 pp
Expected all-buyer effect+0.20 pp
Separate selection example10.20%
Selection-example denominator1,000 assigned buyers

Left: dilution assumes a fixed, treatment-independent affected share and zero effect elsewhere. Right: a separate constructed treatment group has 102 purchasers among 1,000 assigned buyers. Dropping 100 failed-load non-purchasers falsely raises reported conversion to 102/900. These are two distinct illustrations, not one fitted model.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

Should failed new-stack loads be removed from treatment conversion?

Your learning note

Private · just for you