0. Causal Questions and Designs

Potential Outcomes and Experiments

Use assignment to recover an average counterfactual while auditing non-compliance and missing outcomes

Potential Outcomes and Experiments

Randomisation targets assignment effects

If offer assignment $Z_i$ is random, then in expectation

Z_i\perp\{Y_i(1),Y_i(0)\}.

The control mean estimates the treated group’s missing mean under no offer, and vice versa. Randomisation balances potential outcomes in expectation, not necessarily every covariate in one realised sample.

Worked difference in means

Pathways randomises 1,000 applicants:

Assignment	Applicants	Completed degree	Rate
offer	500	290	58%
no offer	500	250	50%

The ITT estimate is $0.58-0.50=0.08$ , or 8 percentage points. A simple standard error is

SE=\sqrt{\frac{0.58(0.42)}{500}+\frac{0.50(0.50)}{500}} \approx0.031.

A rough 95% interval is about 1.8 to 14.2 points. The point estimate is not “the true effect”; it is one random assignment’s estimate of a defined mean effect.

Random assignment versus self-selection

The self-selected comparison combines the treatment effect with pre-existing ability differences. Running a more elaborate model does not change how treatment was assigned.

Non-compliance separates offer from receipt

Suppose an offer raises scholarship receipt from 20% to 70%, a 50-point first stage, while completion rises by 8 points. The Wald ratio is

\frac{0.08}{0.70-0.20}=0.16.

Under IV assumptions, this is a 16-point LATE for applicants whose receipt is changed by the offer. It is not the ATE of scholarship receipt for everyone.

Report both:

ITT: effect of the policy lever—being offered;
first stage: how much the offer changed receipt;
receipt effect: only under the additional IV argument.

Attrition can undo assignment comparability

If outcome data are observed for 95% of controls but 80% of offered applicants, the observed groups need not remain comparable. Report follow-up by assignment, reasons, baseline predictors and outcome bounds or justified adjustments.

Do not condition only on “students found in the graduate database” if treatment changes database coverage.

Spillovers and assignment level

If an offer is randomised by school, uncertainty must reflect school-level assignment. Treating thousands of pupils as independently assigned produces falsely narrow intervals. Peer spillovers may make the relevant estimand a school-level policy effect rather than an individual direct effect.

Experiment audit

Item	Question
assignment	was the random sequence implemented and concealed?
analysis	is the primary estimate intention-to-treat?
balance	are large chance imbalances understood, not significance-shopped?
attrition	is missingness reported by assignment?
interference	can one unit’s assignment affect another?
outcome	was it defined before looking at results?
multiplicity	which outcome and subgroup claims were primary?

Quick check

The randomised estimate is imprecise, while an adjusted observational estimate is very precise. Which is more credible?

Answer

Precision and identification are different. Randomisation may provide a stronger counterfactual but wide uncertainty; the observational estimate may be biased despite narrow standard errors. Report both design credibility and sampling uncertainty rather than ranking them by p-value.

Next: Causal Diagrams and Controls

Edit this pageorReport an issue

Causal Questions and Estimands

Translate a policy question into treatment versions, potential outcomes and a target population

Causal Diagrams and Controls

Distinguish confounders, mediators, colliders and proxies before adjustment