Module 4 — Estimation, Likelihood, and Inference

Core question

How does a finite dependent vector identify model parameters, and how should its covariance structure enter estimation and uncertainty?

Learning outcomes

You will be able to:

• express conditional AR estimation as matrix least squares;
• derive Yule–Walker estimators from autocovariance equations;
• evaluate an exact Gaussian likelihood with Cholesky factors;
• connect the joint likelihood to the innovations likelihood;
• obtain curvature-based standard errors and state their limitations;
• diagnose a fitted model with residual covariance, not one scalar score.

1. Conditional AR estimation is regression

For an AR($p$),

stack observations $t=p+1,\ldots,n$:

Then

This is conditional least squares because the first $p$ observations are treated as fixed initial conditions. In R, qr.solve(X, y) or lm.fit(X, y) is preferable to forming $(X^\top X)^{-1}$.

2. Yule–Walker estimates second-order equations

For a stationary AR($p$), multiply the model by $X_{t-h}$ and take expectations. For $h\ge1$:

The first $p$ equations are

Replacing population autocovariances by sample autocovariances gives

Moment and conditional estimators need not agree in a short sample. Their difference is evidence about finite-sample conventions, not automatically an error.

3. Exact Gaussian likelihood uses the whole covariance matrix

Suppose

The log likelihood is

Every term has a role:

For an AR(1) with zero mean,

Near $|\phi|=1$, this matrix becomes increasingly ill-conditioned. That is both a statistical boundary and a numerical warning.

4. Cholesky evaluation is the stable route

If $\Sigma=U^\top U$ with upper-triangular $U=\operatorname{chol}(\Sigma)$, then

and

The computation uses triangular solves and never forms $\Sigma^{-1}$.

R: exact AR(1) matrix likelihood

This cell simulates a centred AR(1), optimises the exact finite-sample likelihood, and compares it with R's arima(..., method = "ML"). Small differences arise from parameterisation and numerical optimisation, but both use the initial-state information that conditional least squares omits.

5. Joint and innovations likelihoods are two factorizations

The joint density factors as

For a Gaussian process, each conditional density is determined by an innovation $v_t$ and variance $F_t$:

They are mathematically equivalent under the same initialisation. R's exact arima likelihood uses a state-space representation and Kalman innovations.

6. Curvature gives local uncertainty

Let

be Fisher information. Under regularity conditions,

The inverse observed Hessian estimates local parameter covariance. Do not interpret it mechanically near unit roots, non-invertible boundaries, unidentified cancellations, or structural breaks: regular asymptotics can fail exactly where the likelihood is flat or irregular.

7. Residuals estimate innovations, not truth

For residual vector $\widehat{\boldsymbol\varepsilon}$, inspect

A well-specified linear mean model should leave off-diagonal entries small relative to sampling variation. Also check:

• changing variance over time;
• outliers and heavy tails;
• parameter sensitivity to endpoints;
• roots close to the unit circle;
• near cancellation of AR and MA polynomials.

The Ljung–Box statistic compresses several sample residual correlations:

A large p-value says that this test did not detect residual autocorrelation at the chosen lags. It does not establish independence, normality, or parameter constancy.

8. Model comparison remains conditional

For maximised log likelihood $\ell$ and $k$ estimated parameters,

Compare values only when models use the same response, likelihood convention, and effective sample. Before reading the smaller number, check admissible roots, identifiability, and residual covariance.

Practice

1. Write $\mathcal X$ and $\mathbf y$ for an AR(2) observed at $x_1,\ldots,x_6$.
2. Derive the AR(1) Yule–Walker estimator.
3. Explain why $\log|\Sigma|$ cannot be dropped when comparing covariance parameters.
4. Show how the Cholesky factor computes $\mathbf x^\top\Sigma^{-1}\mathbf x$.
5. Why can Hessian standard errors be misleading when $\widehat\phi$ is close to one?

Takeaway

Least squares, moments, and likelihood are different matrix summaries of the same dependent vector. Their assumptions, initial conditions, and numerical factorizations must be visible. Continue to Multivariate Systems.