Classification and Calibration
Classification and Calibration
Start with the confusion matrix
HarborMart has 10,000 eligible customers; 300 become inactive. At one threshold, a model flags 500 customers, including 180 who become inactive.
| Observed inactive | Observed active | Total | |
|---|---|---|---|
| flagged | TP 180 | FP 320 | 500 |
| not flagged | FN 120 | TN 9,380 | 9,500 |
| total | 300 | 9,700 | 10,000 |
Therefore:
Accuracy is 95.6%, below the 97% accuracy of predicting “active” for everyone. Class imbalance makes accuracy a weak headline.
Ranking and probability are different
- discrimination: do higher scores belong to more positive cases? AUROC and precision–recall assess ranking.
- calibration: among cases assigned probability 0.20, does the outcome occur about 20% of the time?
A monotonic transformation can preserve ranking and destroy probability meaning. Decision thresholds using expected value require calibrated probabilities.
Calibration table
| Predicted-risk band | Customers | Mean predicted | Observed inactivity |
|---|---|---|---|
| 0–5% | 6,000 | 2% | 0.5% (30) |
| 5–15% | 2,500 | 9% | 1.6% (40) |
| 15–30% | 1,000 | 22% | 5.0% (50) |
| above 30% | 500 | 44% | 36.0% (180) |
The observed counts sum to the same 300 inactive customers as the confusion matrix. The model ranks useful cases but overstates absolute risk, so an ROI calculation that treats its probabilities as literal will be optimistic.
The Brier score is
It assesses probability error but depends on prevalence; compare it with a base-rate forecast and inspect calibration by time and subgroup.
Precision–recall under rare outcomes
When positives are rare, false positives can dominate operations even with a low false-positive rate. Precision directly answers “of those flagged, how many were positive?” The random precision baseline equals prevalence.
Saito and Rehmsmeier (2015) explain why precision–recall plots often reveal practical weakness that an ROC plot obscures under imbalance.
Labels are measurements
“Churn” might mean no order for 30, 60 or 90 days. Customers may be inactive because HarborMart stopped serving their postcode. Label windows must mature, and policy-driven labels can feed past decisions back into the model.
Audit:
- definition and horizon;
- censoring and delayed outcomes;
- class-specific label error;
- whether action changes the observed label;
- whether the target is a proxy for the real harm.
Quick check
Two models have the same AUROC. Model A is well calibrated; Model B overstates every probability by roughly 50%. Are they interchangeable?
Answer
Foundation
Guo et al. (2017) show that strong classification accuracy does not guarantee well-calibrated confidence and evaluate post-processing calibration methods.