Inference and Clustering
Inference and Clustering
Ask what could have varied
A standard error describes variation under a reference process. Candidates include:
- repeated sampling of units from a population;
- repeated random assignment of a fixed sample;
- repeated shocks correlated within schools, firms or regions;
- model-based disturbances under stated assumptions.
Choosing robust=True without naming the reference process is not an inference plan.
Heteroskedasticity versus clustering
Heteroskedasticity-robust standard errors allow observation-specific variances but still require suitable independence across observations. Cluster-robust errors allow arbitrary dependence within clusters and rely on many sufficiently independent clusters.
If Pathways is assigned at school level, pupils in a school share assignment and shocks. The effective assignment count is the number of schools, not the number of pupils.
Worked scale comparison
Suppose 1,000 pupils are nested in 40 schools. With an intra-school correlation of 0.10 and 25 pupils per school, the approximate design effect is
Treating pupils as independent can understate variance by roughly this factor in a simple equal-cluster setting. This calculation is diagnostic, not a substitute for design-appropriate estimation.
Cluster where assignment or residual dependence lives
| Design | Likely inference concern |
|---|---|
| pupil lottery within each school | strata-aware individual assignment; possible school dependence |
| policy assigned to 40 schools | school-level assignment and clustering |
| municipality policy over 12 years | serial correlation within municipality |
| national policy with one treated country | conventional many-cluster asymptotics are unavailable |
| RDD at an individual score cutoff | local observations; account for design and bandwidth procedure |
Clustering at the finest available level is not automatically conservative or valid. The level follows the assignment and sampling structure.
Abadie et al. (2023) formalise sampling and design reasons for clustering and show why the decision is more subtle than “observations share a label.”
Few clusters require different evidence
With few treated or total clusters, ordinary cluster-robust approximations can perform poorly. Depending on the design, consider:
- randomisation or permutation inference using the actual assignment;
- wild-cluster bootstrap with a justified implementation;
- small-sample degrees-of-freedom corrections;
- aggregation to the assignment level;
- honest acknowledgement that precision is weak.
No correction creates independent treatment variation that the study lacks.
Confidence intervals and substantive uncertainty
A 95% interval does not contain every uncertainty source. It normally excludes:
- violation of exclusion or parallel-trend assumptions;
- outcome mismeasurement and attrition;
- treatment-version ambiguity;
- specification and outcome searching;
- transport to another population.
Report sampling uncertainty beside design sensitivity rather than allowing one narrow interval to imply total certainty.
A minimum inference note
Standard errors are clustered by school because assignment occurred at school level and outcomes may share school shocks; there are 40 clusters, of which 20 were assigned treatment. We also report randomisation-inference p-values using the blocked assignment rule.
This is more informative than “robust standard errors in parentheses.”
Quick check
A district policy is assigned once to each of eight districts, but the dataset contains 200,000 residents. Can resident-level robust standard errors justify a precise estimate?