Heterogeneity and Policy Learning
Heterogeneity and Policy Learning
An average can hide decision-relevant variation
Let
be the conditional average treatment effect. A Pathways ATE of six points could combine:
| Pre-specified group | Estimated effect | 95% interval | Applicants |
|---|---|---|---|
| limited adviser access | +12 pp | 5, 19 | 300 |
| moderate access | +6 pp | 1, 11 | 500 |
| strong access | +1 pp | −4, 6 | 200 |
This suggests targeting, but noisy subgroup estimates can manufacture a ranking. Heterogeneity requires more information than an ATE.
Confirmatory and exploratory heterogeneity differ
| Confirmatory | Exploratory |
|---|---|
| groups and contrast pre-specified | patterns learned from many covariates |
| limited multiplicity | large search space |
| direct inference possible | honest sample splitting needed |
| tests a theory | generates a theory or policy candidate |
Do not discover “high impact among rural women aged 19–21” in the same sample and report an ordinary confidence interval as if the subgroup had been planned.
Honest trees and forests separate discovery from estimation
A causal tree can use one subsample to choose splits and another to estimate leaf effects. Causal forests average many such trees and can estimate conditional effects with local weighting. Athey, Tibshirani and Wager (2019) develop generalized random forests; the current grf documentation explains estimation and diagnostics.
Audit:
- overlap within important leaves;
- calibration of heterogeneity, not only variable importance;
- stability across seeds and specifications;
- support for extreme CATE predictions;
- out-of-sample validation of policy value.
Variable importance does not establish that a variable causally modifies treatment.
A policy rule needs a value function
Let decide who receives an offer. A simple welfare objective is
where outcomes and costs must be placed on a commensurable scale. Real rules also face capacity and fairness constraints.
If Northbridge has 400 offers, point estimates suggest offering to all 300 limited-access applicants and 100 moderate-access applicants. Before adopting this rule, ask:
- are group effects precise enough to support the ranking?
- how are ties within the moderate group resolved?
- does adviser access encode inequitable prior provision?
- will expansion change take-up, capacity or treatment quality?
- what is the policy’s value on an untouched evaluation sample?
Athey and Wager (2021) develop efficient policy learning under explicit policy-class constraints. The constraint is part of the estimand, not an inconvenience added after estimation.
Effect prediction is not outcome prediction
A model can predict completion accurately using prior attainment yet learn treatment-effect differences poorly. For policy choice, validate uplift or policy value through honest evaluation; ordinary predictive accuracy is insufficient. Nie and Wager (2021) develop the R-learner as one route to estimating heterogeneous effects with flexible nuisance models.
Quick check
The largest predicted CATEs occur where only 2% of applicants received treatment. Should the policy target that region first?