Only some eligible buyers reach a product page served by the changed path.
A large local gain can become a small storewide effect.
That dilution is real for the broad policy question.
Triggering & dilution: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
Concept
What it means
Northstar example
👥Population defines the effect.
The all-assigned estimate asks what happens when we enable the policy for eligible buyers.
A safely triggered estimate asks what happens among buyers who would encounter the affected path.
🛒 Northstar distinguishes the effect of enabling a cache for all eligible buyers from the effect among buyers who would encounter the changed path.
🚪Trigger symmetrically.
Instrument a counterfactual eligibility event in both groups before the treatment changes behavior.
‘Used the new cache’ is not available in control and can depend on whether treatment succeeded.
🛒 Log comparable eligibility in both Northstar arms rather than filtering treatment to buyers whose new-cache load happened to succeed.
🧮A scaling shortcut needs assumptions.
Overall effect ≈ affected share × local effect only when unaffected buyers have zero effect, selection is stable, and the effect measure is compatible.
Interference and treatment-dependent triggering can break it.
🛒 Affected share × local effect is a useful Northstar illustration only when the share is stable and treatment-independent and unaffected buyers truly have zero effect.
Left: dilution assumes a fixed, treatment-independent affected share and zero effect elsewhere. Right: a separate constructed treatment group has 102 purchasers among 1,000 assigned buyers. Dropping 100 failed-load non-purchasers falsely raises reported conversion to 102/900. These are two distinct illustrations, not one fitted model.
Constructed teaching example · not Amazon data or a production experimentation engine