PART IV · Sharper estimates · CONCEPT 15 OF 24
Cap the extremes, keep the people—and change the question.
EXPLORE~7 min
Back to journey- Revenue per buyer is dominated by a few very large baskets.
- Winsorizing revenue at a predeclared cap keeps every buyer but changes the metric to capped revenue per buyer.
| Concept | What it means | Northstar example |
|---|---|---|
| Capping is not deletion. |
| Northstar caps an extreme revenue value at a declared threshold but keeps the buyer in the sample instead of deleting an inconvenient high spender. |
| The estimand changes. |
| A new cache that helps Northstar's largest buyers may have a real uncapped-revenue effect that a capped metric largely removes. |
| Choose the cap without fishing. |
| Choose Northstar's cap using historical evidence or a valid predeclared method, not whichever cap makes the current treatment significant. |
LAB 15
Heavy tails & capping · Northstar
Uncapped treatment revenueCapped treatment revenue
Uncapped mean effect$7.40
Capped mean effect$0.40
Capped control mean SE$3.15
Buyers retained per arm200 / 200
Constructed matched revenue profiles, not randomized observations or an inferential A/B score. Large buyers receive a larger constructed benefit. Lower the cap: the descriptive mean standard error falls, but much of the real uncapped benefit disappears. No treatment confidence interval is claimed.
Constructed teaching example · not Amazon data or a production experimentation engine
A capped revenue win directly proves what?
Your learning note
Private · just for youFurther reading: Microsoft Experimentation Platform ↗NIST statistical handbook ↗