Layout
PART IV · Sharper estimates · CONCEPT 15 OF 24

Cap the extremes, keep the people—and change the question.

EXPLORE~7 min
Back to journey
  • Revenue per buyer is dominated by a few very large baskets.
  • Winsorizing revenue at a predeclared cap keeps every buyer but changes the metric to capped revenue per buyer.
Heavy tails & capping: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Capping is not deletion.
  • Replace values above a cap with the cap; do not remove the buyers.
  • A genuine high spender is not a data error merely because the value is inconvenient.

Northstar caps an extreme revenue value at a declared threshold but keeps the buyer in the sample instead of deleting an inconvenient high spender.

The estimand changes.
  • Lower variance is useful, but the effect on capped revenue is not the effect on total revenue.
  • A treatment that helps large buyers can lose much of its real benefit after capping.

A new cache that helps Northstar's largest buyers may have a real uncapped-revenue effect that a capped metric largely removes.

Choose the cap without fishing.
  • Use historical data or an otherwise valid predeclared method and apply the same cap in both arms.
  • Report uncapped revenue alongside it, and never tune caps until the result becomes significant.

Choose Northstar's cap using historical evidence or a valid predeclared method, not whichever cap makes the current treatment significant.

LAB 15

Heavy tails & capping · Northstar

Interactive time series chart−$0k$1k$2k$3kBuyer 1Buyer 200
Uncapped treatment revenueCapped treatment revenue
Uncapped mean effect$7.40
Capped mean effect$0.40
Capped control mean SE$3.15
Buyers retained per arm200 / 200

Constructed matched revenue profiles, not randomized observations or an inferential A/B score. Large buyers receive a larger constructed benefit. Lower the cap: the descriptive mean standard error falls, but much of the real uncapped benefit disappears. No treatment confidence interval is claimed.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

A capped revenue win directly proves what?

Your learning note

Private · just for you