Validation, Baselines and Leakage
Validation, Baselines and Leakage
Validation is a time machine with rules
At each historical prediction origin, reconstruct only information available then. HarborMart’s late-delivery model is trained on orders whose outcomes have matured; its test set contains later completed weeks.
| Partition | Weeks | Purpose |
|---|---|---|
| train | 1–16 | estimate model parameters |
| validation | 17–20 | choose features, complexity and threshold |
| test | 21–24 | one final performance audit |
If the business has strong seasonality, a single period may be fragile. Use rolling origins and report performance by horizon and regime.
Four leakage routes
| Leakage | HarborMart example | Repair |
|---|---|---|
| target leakage | final refund amount predicts late delivery | exclude post-outcome fields |
| temporal leakage | current lifetime orders in a 2024 row | reconstruct as of origin |
| group leakage | same customer appears in train and random test | group or time-aware split |
| preprocessing leakage | imputation/scaling fitted before split | fit transforms on training only |
Leakage is information unavailable in intended use, not simply a variable highly correlated with the target.
Match the split to deployment
| Deployment question | Suitable validation idea |
|---|---|
| future weeks at known stores | forward-chaining time split |
| new stores | hold out stores |
| new customers | group by customer identity |
| next campaign under changed policy | later-policy test and stress scenario |
| rare disruptions | targeted stress set plus ordinary test |
No split protects against every shift. Name the intended generalisation.
Baselines before algorithms
For classification, compare with:
- observed base rate;
- current business rule;
- simple interpretable score;
- capacity-matched random or priority policy.
For forecasting, compare with:
- last value;
- seasonal naive value;
- recent mean;
- current planner forecast.
A complex model that beats “predict zero” but not the deployed rule has not established incremental value.
Avoid test-set bargaining
Choose model families and hyperparameters on training/validation evidence. Use the test once for the selected pipeline. Repeatedly consulting test performance turns the test into another validation set.
Record every material comparison, including failed models. Otherwise the reported interval ignores the search process.
Evaluation table
| Dimension | Minimum report |
|---|---|
| central performance | MAE, log loss or decision loss |
| uncertainty | interval across periods/folds or bootstrap unit |
| stability | by time, store, channel and relevant group |
| calibration | predicted versus observed risk |
| baseline | same records and metric |
| stress | missing fields, peaks, policy or price shift |
| latency | whether output arrives before action |
Quick check
A delivery model uses “number of support contacts in the next 24 hours.” It will be available before the weekly management meeting. Is it valid?