Classical Time Series — Course Guide

A matrix-first course in covariance-stationary processes, ARMA equations, linear prediction, likelihood, state space, and spectra.

Classical Time Series — Covariance, Operators, and Prediction

Course identity

This is a traditional statistical time-series course for advanced undergraduates and graduate students. Its central object is a stochastic process, not a collection of empirical forecasting recipes. Data examples are short simulations or hand-checkable numerical cases used to verify theory.

For a finite block,

\mathbf X_n=(X_1,\ldots,X_n)^\top,\qquad \boldsymbol\mu_n=E(\mathbf X_n),\qquad \Gamma_n=\operatorname{Cov}(\mathbf X_n),

the course repeatedly asks four questions:

What structure must $\Gamma_n$ have?
Which linear operator generates or transforms $\mathbf X_n$ ?
Which normal equations define an estimator or predictor?
Which matrix recursion propagates uncertainty?

Whenever a scalar formula appears, identify its vector, matrix, and quadratic-form version. The scalar formula builds intuition; the matrix form reveals what generalises.

Preparation

You should already be able to:

calculate expectation, variance, covariance, and conditional expectation;
multiply partitioned matrices and solve $\mathbf A\mathbf x=\mathbf b$ ;
interpret eigenvalues, positive definiteness, and a Cholesky factor;
use likelihood, least squares, and asymptotic standard errors;
read elementary R code.

If stationarity is unfamiliar, begin with the 30-minute preparation note.

Ten-minute diagnostic

Why must every covariance matrix be positive semidefinite?
If $\Gamma_{ij}=\gamma(i-j)$ , what pattern appears along each diagonal?
What does $\rho(F)<1$ mean for powers $F^h$ ?
Why is solving $\Gamma\mathbf a=\mathbf g$ preferable to computing $\Gamma^{-1}$ explicitly?
What does a Kalman update do when an observation is missing?

Answers: variances of all linear combinations are non-negative; the matrix is Toeplitz; the powers decay to zero; linear solves are more stable and efficient; it performs prediction but skips measurement correction.

Learning outcomes

By the end, you will be able to:

construct and test Toeplitz covariance matrices for stationary processes;
derive ARMA autocovariances from difference equations and lag polynomials;
move between AR( $p$ ) equations, companion matrices, roots, and impulse responses;
obtain best linear predictors from projection normal equations;
derive Yule–Walker, conditional least-squares, and Gaussian-likelihood estimators;
express multivariate, seasonal, spectral, and state-space models with block matrices;
implement each derivation in base R and check dimensions, symmetry, eigenvalues, and numerical error;
explain which conclusions require Gaussianity and which use second moments only.

The matrix spine

Statistical idea	Matrix statement	R operation
stationary covariance	$\Gamma_n=[\gamma(i-j)]$	`toeplitz(gamma)`
valid covariance	$\mathbf a^\top\Gamma_n\mathbf a\ge0$	`eigen(..., symmetric=TRUE)`
Gaussian simulation	$\mathbf X=\boldsymbol\mu+L\mathbf z$ , $LL^\top=\Gamma$	`chol` and `%*%`
AR( $p$ ) recursion	$\mathbf s_t=F\mathbf s_{t-1}+G\varepsilon_t$	`eigen(F)` and recursive `%*%`
Yule–Walker	$\Gamma_p\boldsymbol\phi=\boldsymbol\gamma_p$	`solve(Gamma, gamma)`
linear prediction	$\Gamma\mathbf a=\operatorname{Cov}(\mathbf X,Y)$	`solve` and a Schur complement
Gaussian likelihood	$\log	\Sigma_\theta
VAR covariance	$\operatorname{vec}(\Gamma_0)=(I-A\otimes A)^{-1}\operatorname{vec}(\Sigma)$	`kronecker`
Fourier transform	$\mathbf d=F_n\mathbf x$	complex `outer` and `%*%`
Kalman filter	prediction and correction of $(\mathbf a_t,P_t)$	matrix recursion

Course map

Module	Main derivation	Executable check
Preparation: stationarity	stable first and second moments	distinguish deterministic and stochastic change
1. Processes and covariance	Toeplitz $\Gamma_n$ and Wold representation	construct, factor, and simulate a covariance matrix
2. ARMA difference equations	lag polynomials, companion form, Yule–Walker equations	compare roots, eigenvalues, and theoretical ACF
3. Linear prediction	projection equations and state-space recursion	compute predictor weights and error variance
4. Estimation and inference	OLS, moments, exact Gaussian likelihood	optimise a covariance-matrix likelihood
5. Multivariate systems	block companion and Kronecker equations	solve a stationary VAR covariance
6. Spectral analysis	Fourier matrix and spectral density	recover frequencies with a discrete transform
7. R matrix laboratory	four integrated derivation labs	browser-based base R notebook
8. Optional Python appendix	empirical implementation contrast	forecasting workflow

The Python appendix is supplementary. It is useful for comparing software conventions, but it is not the conceptual spine of this course.

A finite-block view of one process

Weak stationarity means that every finite block has

E(\mathbf X_n)=\mu\mathbf 1_n, \qquad \Gamma_n= \begin{bmatrix} \gamma(0)&\gamma(1)&\cdots&\gamma(n-1)\\ \gamma(1)&\gamma(0)&\cdots&\gamma(n-2)\\ \vdots&\vdots&\ddots&\vdots\\ \gamma(n-1)&\gamma(n-2)&\cdots&\gamma(0) \end{bmatrix}.

This one matrix connects the course:

ARMA equations restrict its entries;
prediction partitions it;
Gaussian likelihood evaluates its determinant and quadratic form;
the spectrum diagonalises its large-sample analogue;
state-space recursions avoid constructing it all at once.

How to read every R example

Each webR cell is self-contained. Read it in this order:

Dimensions: write the size of every vector and matrix.
Identity: name the equation the code implements.
Numerical check: inspect symmetry, eigenvalues, or reconstruction error.
Perturbation: change one parameter and predict the direction of change first.
Conclusion: state a mathematical claim, not merely “the code ran.”

Example conclusion:

The smallest eigenvalue of $\Gamma_6$ is positive, so this six-dimensional covariance block is positive definite. As $\phi$ moves from $0.6$ to $0.95$ , the condition number increases; near-unit-root covariance matrices are harder to distinguish numerically.

Ten-week route

Week	Preparation	Seminar or laboratory output
1	random vectors and covariance	build a Toeplitz covariance by hand
2	stationarity and Wold representation	prove a covariance sequence is admissible or find a counterexample
3	AR, MA, and lag polynomials	convert an AR( $p$ ) to companion form
4	Yule–Walker equations	recover AR coefficients from autocovariances
5	Hilbert-space projection	derive a best linear predictor and its error variance
6	estimation and Gaussian likelihood	compare conditional and exact estimators
7	VAR and Kronecker products	solve a discrete Lyapunov equation
8	Fourier representation	connect a periodogram to a Fourier matrix
9	state-space models	implement Kalman prediction and correction
10	synthesis	defend every line of one R matrix computation

Assessment alignment

Task	Weight	Evidence expected
derivations	35%	correct assumptions, dimensions, and algebra
R matrix labs	30%	reproducible code plus numerical verification
theory critique	15%	identify what fails without stationarity or Gaussianity
final synthesis	20%	connect covariance, prediction, likelihood, and state space

Graduate extensions ask for proofs, asymptotic arguments, multivariate identification, or computational complexity. Undergraduate solutions may use stated theorems but must still explain dimensions and assumptions.

Reading ladder

Reading	Use in this course
Brockwell & Davis, Time Series: Theory and Methods	stationary processes, Hilbert-space prediction, ARMA, spectra, and asymptotics
Durbin & Koopman, Time Series Analysis by State Space Methods	matrix filtering, smoothing, likelihood, and initialisation
R `stats::ARMAacf` documentation	implementation of theoretical ARMA autocorrelations from difference equations
R `stats::KalmanLike` documentation	the state-space machinery used by R's exact ARIMA likelihood
Düker et al. (2024/25), VARMA review	graduate bridge from classical matrix equations to modern identification and estimation
Zheng (2024/25), infinite-order high-dimensional VAR	recent example of preserving interpretable linear dynamics under high dimensionality

The first four readings establish the course. The recent papers show where the same matrix ideas lead; they do not replace the foundations.

Start

Complete the stationarity preparation, then open Module 1. Keep one page of matrix dimensions beside every derivation.

Edit this pageorReport an issue

Portfolio Risk Capstone

Integrate reserving, frequency, severity, reinsurance, aggregate capital, and ruin in one auditable recommendation.

Preparation — Stationarity in 30 Minutes

A compact diagnostic primer on stable dependence, trends, unit roots, seasonality, and breaks.