Layout
PART V · Hidden traps · CONCEPT 17 OF 24

An effect on one page can disappear in the whole store.

EXPLORE~7 min
Back to journey
  • Only some eligible buyers reach a product page served by the changed path.
  • A large local gain can become a small storewide effect.
  • That dilution is real for the broad policy question.
Triggering & dilution: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Population defines the effect.
  • The all-assigned estimate asks what happens when we enable the policy for eligible buyers.
  • A safely triggered estimate asks what happens among buyers who would encounter the affected path.

Northstar distinguishes the effect of enabling a cache for all eligible buyers from the effect among buyers who would encounter the changed path.

Trigger symmetrically.
  • Instrument a counterfactual eligibility event in both groups before the treatment changes behavior.
  • ‘Used the new cache’ is not available in control and can depend on whether treatment succeeded.

Log comparable eligibility in both Northstar arms rather than filtering treatment to buyers whose new-cache load happened to succeed.

A scaling shortcut needs assumptions.
  • Overall effect ≈ affected share × local effect only when unaffected buyers have zero effect, selection is stable, and the effect measure is compatible.
  • Interference and treatment-dependent triggering can break it.

Affected share × local effect is a useful Northstar illustration only when the share is stable and treatment-independent and unaffected buyers truly have zero effect.

LAB 17

Triggering & dilution · Northstar

Local absolute effect (supplied)+0.80 pp
Expected all-buyer effect+0.20 pp
Separate selection example10.20%
Selection-example denominator1,000 assigned buyers

Left: dilution assumes a fixed, treatment-independent affected share and zero effect elsewhere. Right: a separate constructed treatment group has 102 purchasers among 1,000 assigned buyers. Dropping 100 failed-load non-purchasers falsely raises reported conversion to 102/900. These are two distinct illustrations, not one fitted model.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

Which trigger is most defensible?

Your learning note

Private · just for you