STOP 08 / 31How wrong—and in what way?
Layout
STOP 08 · Measuring

How wrong—and in what way?

EXPLORE~6 min
Back to route map
  • Northstar needs to know both typical error and the cost of major misses.
  • No single score tells the whole story, especially when a holiday weekend matters far more than an ordinary Tuesday.
Forecast accuracy and direction. Illustrative Northstar examples, not measured business results.
MetricWhat it meansNorthstar example
MAE

Mean absolute error: average size of the miss in the original units.

Interpret Northstar's miss in dollars or index points, but do not compare raw MAE across businesses of very different sizes.

RMSE

Square errors, average them, then take the square root; large misses get more influence.

  • One very large Black Friday miss can dominate Northstar's RMSE.
  • That may match business costs—or hide typical-day behavior.
MAPE

Mean absolute error as a percentage of each actual observation.

Near-zero Northstar sales produce extreme percentages; an actual of zero makes the ordinary formula undefined.

WAPE

Sum of absolute errors divided by sum of absolute actuals.

  • Larger Northstar sales months contribute more to this aggregate score.
  • It is still undefined when aggregate actuals are zero.
MASE

Test MAE divided by a naïve error scale computed on training history.

  • Below one means Northstar's test MAE is smaller than that training scale.
  • It is not automatic proof of beating a naïve forecast on the same holdout.
Bias

Mean signed error, using actual minus forecast.

  • Positive means Northstar underpredicts on average.
  • Opposite misses can cancel, so pair bias with an accuracy score.
error = actual − forecast
FIG 08.1

One big miss changes the score

WHAT THIS SHOWS

Compare Northstar's supplied monthly forecasts with simulated actual sales. Change systematic error, zero-mean noise, one large miss, and a zero-sales month to see which metrics react.

The question it answers: Do you miss by a lot, systematically in one direction, or mainly on a few extreme/low-sales months?

081161123456789101112Monthly sales index · black: actual · red: forecast · gray: miss distance
MAE · index points5.11
RMSE · index points5.63
MAPE4.51%
WAPE4.42%
MASE0.71
Bias · actual − forecast-0.00
Fine print & method

These are supplied forecast perturbations, not fitted production predictions. Noise is centered to have zero average additive effect. MAPE is undefined at a zero actual; WAPE is finite here because aggregate actual sales remain positive, not because it always solves zero-denominator problems. MASE divides test MAE by a one-step naïve MAE computed from separate supplied training observations. Positive bias means underprediction.

CHECK YOURSELF · CHOOSE AN ANSWER

Which metric penalizes a rare huge miss more strongly?

Your learning note

Private · just for you