Layout
PART V · Hidden traps · CONCEPT 18 OF 24

More orders can make revenue per order fall.

EXPLORE~7 min
Back to journey
  • Faster pages may encourage extra small purchases.
  • Revenue per order could fall while revenue per buyer rises.
  • Ratios can hide changes in their own denominators.
Ratio metrics: concepts and Northstar examples. Illustrative Northstar examples, not measured business results.
ConceptWhat it meansNorthstar example
Ratio of totals is not mean of ratios.
  • Total revenue / total orders weights frequent orderers differently from an average of buyer-level revenue/order ratios.
  • Buyers with no orders also make the latter undefined.

Northstar's total revenue divided by total orders is not an unweighted average of each buyer's revenue/order ratio, especially for zero-order buyers.

Show numerator and denominator.
  • Report revenue, orders, and assigned buyers together.
  • Revenue per buyer includes zero-purchase buyers and targets a different business quantity from average order value.

If faster pages generate more small purchases, Northstar's average order value can fall while revenue per assigned buyer rises.

Inference must preserve covariance.
  • Use a suitable delta-method or buyer-cluster bootstrap analysis for a ratio of totals.
  • Treating the denominator as fixed or requests as independent can misstate uncertainty.

A buyer-cluster bootstrap can retain the relationship between Northstar's revenue and order counts; treating the denominator as fixed misses that covariance.

LAB 18

Ratio metrics · Northstar

Control · orders / AOV100 / $50
Treatment · total revenue$5,400
AOV relative change-10.0%
Revenue/buyer relative change8.0%

Both arms have 1,000 assigned buyers. Control revenue is $5,000. Increase treatment orders while lowering their average value: revenue per buyer and average order value can move in opposite directions. These aggregate totals do not provide the buyer-level covariance needed for a confidence interval.

Constructed teaching example · not Amazon data or a production experimentation engine
CHECK YOURSELF · CHOOSE AN ANSWER

Average order value falls while revenue per buyer rises. Is that impossible?

Your learning note

Private · just for you