Module 1 — Processes, Dependence, and Stationarity

Core question

We observe one sequence $x_1,\ldots,x_n$. What assumptions let that one path reveal a population mean, variance, and dependence structure?

Learning outcomes

You will be able to:

• distinguish a stochastic process, a random variable, and one sample path;
• construct the mean vector and Toeplitz covariance matrix of a stationary finite block;
• calculate and interpret autocovariance and autocorrelation;
• separate strict stationarity, weak stationarity, and ergodicity;
• distinguish white noise from independence and Gaussianity;
• derive the dependence and shock persistence of a stationary AR(1);
• state exactly what the Wold representation does and does not promise.

1. Process versus path

A stochastic process is a collection of random variables indexed by time:

The capital letters describe the population mechanism. The data $x_1,\ldots,x_n$ are one realised path. Observing 120 months is not the same as observing 120 independent copies: adjacent values may share shocks.

2. Finite blocks expose the matrix structure

For any selected times $t_1,\ldots,t_n$, define

Its first two moments are

For consecutive observations from a weakly stationary process,

$\Gamma_n$ is symmetric Toeplitz: each diagonal is constant. It must also be positive semidefinite because

for every $\mathbf a$. The bounds $|\rho(h)|\le1$ are necessary but not sufficient; all lags must fit together into valid covariance matrices.

Change $\phi$ to $-0.7$: alternating covariances appear, but validity remains. Move it to $0.98$: the smallest eigenvalue and condition number reveal near-collinearity even before estimation.

3. Dependence is indexed by lag

For a weakly stationary process,

$\gamma(h)$ retains the unit squared; $\rho(h)$ is unit-free and lies between $-1$ and $1$. Neither measures nonlinear dependence completely.

Worked sample calculation

Take one short path:

The deviations are $(-2,0,-1,1,0,2)$. Using divisor $n$,

so $\widehat\rho(1)=-0.10$. The sample is far too short for a strong conclusion; the exercise teaches what each ACF bar aggregates.

4. Three kinds of stability

Strict stationarity implies weak stationarity only when second moments exist. For a Gaussian process, weak stationarity is enough to determine shift-invariant finite-dimensional distributions. Stationarity alone does not guarantee that one path explores the population adequately; that is the role of ergodic conditions.

Why persistence reduces effective information

For a stationary mean-zero series,

Positive autocovariances make the sample mean noisier than the iid formula $\gamma(0)/n$. One hundred highly persistent months need not contain one hundred months' worth of independent information.

5. White noise is not one assumption

Uncorrelated does not imply independent outside special families such as jointly Gaussian variables. A squared-noise series can be predictable in variance even when the level has zero autocorrelation.

6. The AR(1) as a complete dependence example

Let

If $|\phi|<1$, repeated substitution gives the causal representation

Therefore

With $\phi=0.7$ and $\sigma_\varepsilon^2=4$:

• long-run variance is $4/(1-0.49)=7.84$;
• lag-3 correlation is $0.7^3=0.343$;
• a unit shock has about half its effect after $\log(0.5)/\log(0.7)=1.94$ periods.

If $\phi<0$, signs alternate; if $\phi=1$, shocks do not decay and the process becomes a random walk.

7. Linear processes and Wold's result

A causal linear process has the form

with coefficients that decay sufficiently for the sum to exist. Its autocovariance is

Wold's decomposition says that every covariance-stationary, purely non-deterministic process can be represented as an infinite moving average of uncorrelated innovations. It does not say that a short finite ARMA model is true, that innovations are independent, or that the coefficients are easy to estimate.

8. Diagnostic distinctions

Practice

1. For $X_t=-0.5X_{t-1}+\varepsilon_t$ with innovation variance 3, calculate $\operatorname{Var}(X_t)$ and $\rho(1),\rho(2),\rho(3)$.
2. Construct two dependent variables with zero correlation; explain why an ACF cannot detect their dependence.
3. For the six-point path above, recompute $\widehat\gamma(1)$ using divisor $n-1$.
4. Explain why stationarity and ergodicity answer different questions.
5. Construct $\Gamma_3$ for an AR(1) with $\phi=0.5$ and innovation variance $0.75$, then find $\operatorname{Var}(X_1-X_2)$ as a quadratic form.

Takeaway

Time-series inference begins by stating what is random, what is observed once, and which form of stability connects one to the other. Continue to ARMA and ARIMA to encode that stable dependence with lag polynomials.