Experiments and Causal Boundaries
Experiments and Causal Boundaries
The missing outcome
For customer , let be the outcome under a new free-delivery message and under the current message. The individual effect is
but only one potential outcome is observed. Random assignment makes groups comparable in expectation, allowing the difference in group means to estimate an average effect.
HarborMart experiment
| Group | Eligible customers | Paid orders | Conversion |
|---|---|---|---|
| control | 10,000 | 2,400 | 24.0% |
| treatment | 10,000 | 2,520 | 25.2% |
Estimated lift is 1.2 percentage points, or 5% relative to the control rate. The standard error for two independent proportions is approximately
A rough 95% interval is , approximately 0.0 to 2.4 percentage points. “Positive point estimate” and “precisely established business value” are different claims.
Design before significance
| Design item | HarborMart choice | Why it matters |
|---|---|---|
| unit of randomisation | customer account | avoid a customer seeing both messages |
| eligibility | customers with a serviceable address | preserve target population |
| primary outcome | contribution per eligible customer | includes discount cost and conversion |
| guardrails | lateness, refunds, support contacts | catch displaced harm |
| horizon | 28 days plus delayed refund window | include foreseeable effects |
| analysis | intention to treat | preserve randomisation |
| stopping rule | fixed or pre-specified sequential plan | avoid repeated peeking bias |
First run an A/A or instrumentation check. A sample-ratio mismatch, duplicate assignment or missing outcome can invalidate a very small p-value.
Interference and marketplace effects
If treatment increases demand, control customers may face fewer slots. One customer’s treatment then changes another customer’s outcome. Randomising individual customers does not isolate the marketplace effect.
Possible responses include cluster or switchback randomisation, capacity-aware designs and explicit equilibrium analysis. The design must match how units interact.
Airbnb’s 2025 experimentation review illustrates current work on long-run ranking dynamics and batch-adaptive experiments. It is an organisational research case, not proof that one design fits every marketplace.
When randomisation is unavailable
Before/after and treated/untreated comparisons can be confounded by season, targeting, selection and concurrent changes. Use domain-supported designs—difference-in-differences, discontinuities, instruments, matching or synthetic controls—only when their identifying assumptions are defensible.
Write the boundary:
“The adjusted association is useful for forecasting and prioritising a future experiment; it does not identify the effect of sending the offer.”
Quick check
Treatment raises conversion but lowers contribution per eligible customer. Is the experiment a success?
Answer
Foundation
- Kohavi et al., Controlled experiments on the web
- Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments