PART V · Hidden traps · CONCEPT 18 OF 24
More orders can make revenue per order fall.
EXPLORE~7 min
Back to journey- Faster pages may encourage extra small purchases.
- Revenue per order could fall while revenue per buyer rises.
- Ratios can hide changes in their own denominators.
| Concept | What it means | Northstar example |
|---|---|---|
| Ratio of totals is not mean of ratios. |
| Northstar's total revenue divided by total orders is not an unweighted average of each buyer's revenue/order ratio, especially for zero-order buyers. |
| Show numerator and denominator. |
| If faster pages generate more small purchases, Northstar's average order value can fall while revenue per assigned buyer rises. |
| Inference must preserve covariance. |
| A buyer-cluster bootstrap can retain the relationship between Northstar's revenue and order counts; treating the denominator as fixed misses that covariance. |
LAB 18
Ratio metrics · Northstar
Control · orders / AOV100 / $50
Treatment · total revenue$5,400
AOV relative change-10.0%
Revenue/buyer relative change8.0%
Both arms have 1,000 assigned buyers. Control revenue is $5,000. Increase treatment orders while lowering their average value: revenue per buyer and average order value can move in opposite directions. These aggregate totals do not provide the buyer-level covariance needed for a confidence interval.
Constructed teaching example · not Amazon data or a production experimentation engine
Average order value falls while revenue per buyer rises. Is that impossible?
Your learning note
Private · just for youFurther reading: Microsoft Experimentation Platform ↗NIST statistical handbook ↗