Trees and Ensembles
Trees and Ensembles
A tree partitions cases
A shallow HarborMart tree might produce:
| Rule | Historical late rate | Support |
|---|---|---|
| slot before 17:00 | 4% | 8,400 orders |
| slot after 17:00 and utilisation ≤85% | 11% | 2,100 |
| slot after 17:00 and utilisation >85% | 32% | 600 |
Each terminal node estimates an outcome from cases following its path. The rules are readable, but the 32% is not a law of nature: report support, uncertainty and future validation.
How a split is chosen
Classification trees select splits that reduce impurity, such as Gini impurity
Regression trees reduce squared or absolute error. Greedy splitting searches locally; a small data change can produce a different tree.
Control complexity with maximum depth, minimum leaf size and pruning chosen on validation data. Growing a tree until every training leaf is pure memorises noise.
Why ensembles help
| Method | Main idea | Typical strength | Main caution |
|---|---|---|---|
| bagging/random forest | average diverse trees | reduces variance, robust baseline | less transparent; correlated trees add less |
| gradient boosting | sequentially improve residual/error direction | strong tabular prediction | sensitive to tuning and leakage |
| stacking | combine out-of-fold model predictions | exploits complementary errors | base predictions must be genuinely out of fold |
Complexity is justified by future performance and decision value, not by algorithm prestige.
Compare on the same information
A fair benchmark holds constant:
- training and test rows;
- feature availability cutoff;
- preprocessing and missingness policy;
- target and horizon;
- metric, threshold and capacity;
- uncertainty calculation.
Giving the ensemble post-outcome features while the logistic model uses checkout-time fields is not a model comparison.
Feature importance is not an effect
Split counts, permutation importance and SHAP-style attributions describe model dependence under specific perturbations. Correlated features can share or exchange importance; a high-importance feature is not necessarily causal or safe to intervene on.
If postcode is important, the next question is whether it encodes distance, service design, socioeconomic structure or data quality—not “change the customer’s postcode.”
Quick check
A boosted model improves AUROC from 0.79 to 0.80 but doubles latency and worsens high-risk calibration. Should it replace the baseline?