- A forecast must be tested on observations that were not available when it was made.
- Time flows forward: a random train/test split can let tomorrow leak into yesterday.
| Evaluation | What it means | Northstar example |
|---|---|---|
| Single holdout | Reserve the later observations; fit on the earlier ones and forecast the reserved period. |
|
| Rolling backtest | Move the origin forward, refit using only the available past, and repeat the same horizons. |
|
Walk forward through time
Pretend it is an earlier month at Northstar. Fit using only history up to that date, forecast the next three months, then compare with the held-out observations. Play repeats the process month by month.
The question it answers: How does a forecasting rule perform on observations it had not yet seen?
Average absolute percentage error · 1 test per horizon
Dec 2024 · 1 origins · 3 held-out forecasts
Fine print & method
Each origin refits the same log-trend/month-indicator regression on its own training prefix. Actual future values are used only for scoring. Errors are pooled by horizon, not fabricated or forced to increase monotonically. MAPE is well-defined for this strictly positive teaching index; it is unsafe at zero. The model is deliberately simple and can lag the constructed accelerating growth. Overlapping horizons make errors dependent.