STOP 27 / 31Could you really know that then?
Layout
STOP 27 · In practice

Could you really know that then?

EXPLORE~7 min
Back to route map
  • A predictor is useful only if it is available when the forecast is made.
  • Realized ad clicks, future stockouts, or next week’s actual conversion rate can make a backtest look magical—and fail in production.
Which future drivers can Northstar use?. Illustrative Northstar examples, not measured business results.
Driver typeWhat it meansNorthstar example
Known in advance

Calendar events and plans recorded at the forecast origin may be legitimate inputs.

  • Northstar knows the next Black Friday date and its currently planned discount.
  • Plans can change, so record their as-of date.
Must be forecast

Unknown future inputs add their own uncertainty to the final sales prediction.

Northstar can use a visitor-traffic forecast or traffic scenarios, not future realized visitors hidden inside a backtest.

Future-outcome leakage

Realized future outcomes and preprocessing that uses later observations cross the information boundary.

Using actual future ad clicks or a centered average that needs future sales can flatter Northstar's test score without producing a usable forecast.

FIG 27.1

Would you know it at forecast time?

WHAT THIS SHOWS

Northstar's daily orders are related to visitor traffic. A regression can use forecast visitors, but it cannot know the realized visitors of future days. Compare honest supplied visitor forecasts with the deliberately leaky alternative.

The question it answers: How does uncertainty in an outside driver enter Northstar's sales forecast?

Training: visitors and daily orders

102029Visitor traffic · thousandsDaily orders

Next 14 days · driver uncertainty included

Northstar orders forecast using an external traffic driver92.3196.6300.9405.2Forecast originHistoryToday+14 days
Actual · evaluation onlyForecast from supplied visitor estimatesPrediction intervals
Leaky MAE · actual future visitors7.9
Honest MAE · forecast visitors41.5
Displayed band half-width · orders±41.2
Fine print & method

Regression coefficients and residual spread use only the first 120 observations. Future driver forecasts are supplied synthetic forecasts with controlled errors, not a fitted traffic forecasting service. Realized future visitors are used only in the explicitly marked leakage option and for evaluation. Bands combine residual variance and supplied driver-error variance under a normal linear approximation; they are not empirically calibrated Northstar business intervals or causal estimates.

CHECK YOURSELF · CHOOSE AN ANSWER

Which is a safe predictor for a forecast made a month before Black Friday?

Your learning note

Private · just for you