STOP 07 / 31Let the future stay unseen.
Layout
STOP 07 · Measuring

Let the future stay unseen.

EXPLORE~7 min
Back to route map
  • A forecast must be tested on observations that were not available when it was made.
  • Time flows forward: a random train/test split can let tomorrow leak into yesterday.
Holdout and rolling-origin evaluation. Illustrative Northstar examples, not measured business results.
EvaluationWhat it meansNorthstar example
Single holdout

Reserve the later observations; fit on the earlier ones and forecast the reserved period.

  • Pretend Northstar has reached December 2024.
  • Fit through December, forecast January–March, and compare with what later happened.
Rolling backtest

Move the origin forward, refit using only the available past, and repeat the same horizons.

  • Repeat Northstar's 1-, 2-, and 3-month-ahead forecasts from January's origin, then February's, and onward.
  • Pool errors separately by horizon.
FIG 07.1

Walk forward through time

WHAT THIS SHOWS

Pretend it is an earlier month at Northstar. Fit using only history up to that date, forecast the next three months, then compare with the held-out observations. Play repeats the process month by month.

The question it answers: How does a forecasting rule perform on observations it had not yet seen?

Evaluation
Rolling-origin Northstar forecast evaluationTraining · past onlyHeld out75.0122.6170.2217.7Origin: Dec 2024Jan 2023Jan 2024Jan 2025Dec 2025
Actual simulated salesForecast from this origin · 1–3 months ahead

Average absolute percentage error · 1 test per horizon

1.05%1 month ahead3.22%2 months ahead1.65%3 months ahead
Pretend it isEnd of Dec 2024
Training months24
Origins evaluated1
Forecasts scored3

Dec 2024 · 1 origins · 3 held-out forecasts

Fine print & method

Each origin refits the same log-trend/month-indicator regression on its own training prefix. Actual future values are used only for scoring. Errors are pooled by horizon, not fabricated or forced to increase monotonically. MAPE is well-defined for this strictly positive teaching index; it is unsafe at zero. The model is deliberately simple and can lag the constructed accelerating growth. Overlapping horizons make errors dependent.

CHECK YOURSELF · CHOOSE AN ANSWER

Which evaluation best matches a 28-day inventory decision?

Your learning note

Private · just for you