Optional Python Lab 4 — Forecast, Backtest, and Monitor

Question

Does the fitted model improve repeated future decisions, or only describe the historical sample elegantly?

The model uses trend, month indicators, and the scheduled intervention as exogenous regressors, with AR(1) residual dynamics. Every rolling origin refits the complete model.

Run the backtest

Interpret with restraint

• Compare the model with seasonal naive at each origin; a single average can hide instability.
• MASE < 1 uses the training sample's seasonal-naive scale. Also inspect direct holdout MAE.
• Twelve observations provide a noisy coverage estimate. Combine many origins and inspect whether misses cluster above or below.
• Model intervals condition on the specified dynamics and exogenous path. They do not include unknown interventions or complete model-selection uncertainty.

Forecast report

Target, origin, and horizons:
Information available at each origin:
Why the intervention path is known:
Benchmark:
Rolling-origin result:
Holdout MAE/MASE:
Nominal interval and empirical coverage:
Regime or month with the largest miss:
Decision supported:
Monitoring/refit trigger:

Stress tests

1. Remove the intervention column from X.
2. Forecast from origin 48, before any post-intervention observations; explain why an all-zero training column cannot estimate the step size.
3. Change the AR coefficient from 0.55 to 0.9 and compare interval growth.
4. Replace month indicators with one sine/cosine pair and measure the parsimony–fit trade-off.

Completion standard

The lab is complete when another student can reproduce the exact origins and explain why no post-origin value entered fitting, transformation, selection, or scaling. Return to the course guide to plan a project or compare the R implementation.