Module 3 — Linear Prediction and State-Space Recursions

Core question

Given a covariance structure and a vector of observations, which linear combination predicts an unobserved value with minimum mean-squared error?

Learning outcomes

You will be able to:

• derive the normal equations for a best linear predictor;
• compute prediction-error variance from a Schur complement;
• explain when a linear predictor equals a conditional expectation;
• obtain multi-step AR predictions from powers of a companion matrix;
• derive the Kalman prediction and correction equations;
• distinguish filtering, prediction, and smoothing.

1. Prediction is a projection problem

Let $Y$ be a centred target and let

For a linear predictor $\widehat Y=\mathbf a^\top\mathbf X$,

Differentiation gives the normal equations

When $\Gamma$ is positive definite,

In computation, use solve(Gamma, g). The inverse notation explains the mathematics; explicitly forming an inverse is usually unnecessary.

2. Orthogonality is the defining check

At the optimum,

The error is therefore orthogonal to every predictor. This statement needs finite second moments, not Gaussianity.

If $(Y,\mathbf X^\top)^\top$ is jointly Gaussian, the conditional mean is linear:

Without joint Gaussianity, this remains the best linear predictor but need not be the best among all nonlinear functions.

3. Worked AR(1) projection

For a stationary AR(1) with variance $\gamma(0)$ and correlation $\rho(h)=\phi^{|h|}$, predict $X_{t+1}$ from

Then

Solving gives $\mathbf a=(0,\phi)^\top$. Once $X_t$ is known, $X_{t-1}$ receives zero extra weight. The matrix calculation recovers the Markov property rather than assuming it.

The error variance is

4. Partitioned covariance and the Schur complement

The joint covariance is

Conditioning removes the part of $Y$ explained linearly by $\mathbf X$. The remaining variance,

is the Schur complement of $\Gamma$. It is non-negative because the joint covariance matrix is positive semidefinite.

R: solve the projection and verify orthogonality

The following cell uses an AR(2). Because the true process uses only two lags, the coefficient on $X_{t-2}$ should be numerically zero when predicting from $(X_t,X_{t-1},X_{t-2})^\top$.

The first two weights reproduce $(\phi_1,\phi_2)$ and the third is zero up to rounding. The MSPE reproduces the innovation variance.

5. Multi-step prediction uses matrix powers

Write an AR($p$) in state form:

With $\mathbf e_1=(1,0,\ldots,0)^\top$,

If $\operatorname{Var}(\varepsilon_t)=\sigma^2$ and $Q=\sigma^2GG^\top$, then

The point prediction propagates the state; the uncertainty accumulates every future innovation.

6. Innovations reveal information one observation at a time

Suppose $\widehat X_{t\mid t-1}$ is the projection on the past. Define the innovation

Innovations are mutually orthogonal. For Gaussian processes they are independent. A joint Gaussian likelihood can therefore be written as a sum of one-step terms:

This is the sequential counterpart of the covariance-matrix likelihood in Module 4.

7. State space avoids a growing covariance inverse

Use the linear Gaussian model

with

If $(\mathbf a_{t-1},P_{t-1})$ summarises the filtered state, prediction is

The observation innovation and its covariance are

Correction is

When $Y_t$ is missing, keep the prediction and skip correction. No ad hoc imputation is required.

Practice

1. For AR(1) with $\phi=0.6$ and $\gamma(0)=4$, construct $\Gamma$ and $\mathbf g$ above and solve for $\mathbf a$.
2. Show that adding a predictor cannot increase the minimum MSPE.
3. For an AR(2), write $F$, $G$, and $Q$ explicitly.
4. Explain why the innovations likelihood and joint covariance likelihood agree for a Gaussian model.
5. In a local-level model, what happens to $P_{t\mid t-1}$ during several consecutive missing observations?

Takeaway

Linear prediction is a covariance problem: solve normal equations, verify orthogonality, and read uncertainty from a Schur complement. State-space recursions perform the same conditioning sequentially. Continue to Estimation and Inference.