Forecast Evaluation for Decisions

Design honest pseudo-out-of-sample tests for point, density, and tail forecasts with long-run covariance and real-time information.

Forecast Evaluation for Decisions

A forecast is an information experiment

A historical forecast $\hat f_{t+h|t}$ is valid only if it can be reconstructed from information available at origin $t$ . This requires:

a training window ending at $t$ ;
the release vintage available at $t$ ;
transformations estimated without future observations;
model and hyperparameter choices made inside the training procedure;
a benchmark and loss fixed before comparing results.

Random train/test splitting is generally inappropriate because it lets later observations help predict earlier ones.

Match the loss to the action

Forecast	Functional	Example loss
conditional mean	$E(y\mid\mathcal F)$	squared error $(y-\hat y)^2$
conditional median	median $(y\mid\mathcal F)$	absolute error $
quantile $\tau$	$q_\tau(y\mid\mathcal F)$	check loss $(\tau-I\{y<q\})(y-q)$
density	full predictive distribution	negative log score or CRPS
volatility	conditional scale/variance	QLIKE or decision-specific loss
VaR and ES	lower-tail risk functionals	joint strictly consistent score

A model can win under RMSE and lose under tail loss. That is not a contradiction; the losses reward different decisions.

Pseudo-out-of-sample design

For origins $t=T_0,\ldots,T-h$ :

reconstruct the data available at $t$ ;
estimate each model on a rolling or expanding window;
issue an $h$ -step forecast;
wait for the target according to a stated vintage convention;
record all model losses at the same origin.

Use an expanding window when parameters are assumed stable and old data remain informative. Use a rolling window when adaptation to structural change is part of the design. Comparing many window lengths on the final test period turns that period into training data.

Comparing two forecasts

Let

d_t=L(y_t,\hat f_{1,t})-L(y_t,\hat f_{2,t}).

Then model 2 has lower average loss when $\bar d>0$ . The Diebold–Mariano test standardises $\bar d$ by an estimate of the long-run variance of $d_t$ :

DM=\frac{\bar d}{\sqrt{\hat S_d/P}},

where $P$ is the number of forecast comparisons. Multi-step and overlapping forecasts require serial-dependence adjustment. A 2026 extension develops joint comparison across multiple horizons rather than selecting one favourable horizon after inspection.

Giacomini and White ask a conditional question: does relative performance vary with the information state? This is often more useful than one full-sample winner when recessions and crises matter.

R laboratory: mean forecasts and tail calibration

HAC forecast comparison and regime-sensitive VaR coverage

A positive DM statistic favours the AR forecast under squared loss in this simulation. For VaR, inspect both regimes: overall coverage can conceal too many exceptions in volatile periods and too few in calm periods.

Tail-risk evaluation

For VaR, test at least:

unconditional coverage: is the exception frequency close to $p$ ?
independence: do exceptions cluster?
conditional coverage: are both frequency and dynamics acceptable?

Expected shortfall cannot be validated by counting a single threshold alone. Joint scoring and regression-based tests use both VaR and ES; Fissler, Ziegel, and Gneiting provide a foundation for comparative backtesting.

Finance versus economics

Design element	Finance	Economics
origin	trade/rebalance timestamp	release date or policy meeting
target	return, volatility, quantile, portfolio loss	growth, inflation, unemployment, recession
real-world constraint	cost, turnover, liquidity, capacity	publication lag, revision, institutional forecast cycle
benchmark	historical mean, random walk, implied measure	AR, no-change, survey, official benchmark
stress analysis	crises, volatility regimes, market closures	recessions, policy regimes, revision episodes
preferred report	statistical loss plus economic value	real-time accuracy plus decision/structural limits

What not to conclude

A lower RMSE does not imply a profitable strategy.
Correct unconditional VaR coverage does not imply independent exceptions.
A forecast winner on revised data need not have won in real time.
A forecast comparison does not identify a causal mechanism.
One favourable horizon among many is not confirmatory evidence unless multiplicity was handled.

Practice

Why must two models be scored on the same origins?
What does $\bar d>0$ mean under the stated definition?
When should a rolling window be preferred to an expanding window?

Answers

Otherwise differences can reflect unequal evaluation periods rather than forecast quality.
Model 2 has lower average stated loss.
When parameter adaptation is part of the prior design and discarding old regimes is substantively defensible.

Next: Integrated R Laboratory.

Edit this pageorReport an issue

State Space, Mixed Frequency, and Nowcasting

Use Kalman prediction and update matrices for latent financial states, macroeconomic nowcasts, missing releases, and ragged edges.

Integrated R Laboratory

Two self-contained browser labs for financial tail risk and real-time macroeconomic forecasting.