Explainability, Monitoring and Drift
Explainability, Monitoring and Drift
Explain the right object
| Object | Question |
|---|---|
| data | where did this feature come from and when was it available? |
| model | which inputs changed this score under the fitted function? |
| uncertainty | how stable is the score across samples or plausible models? |
| decision | why did this score, threshold and constraint produce this action? |
| outcome | did the action improve the intended result? |
A local feature attribution answers none of the other four automatically.
Three levels of explanation
- global: broad model structure, important inputs, interactions and limits;
- cohort: performance and behaviour for an operational or affected group;
- local: evidence and rules behind one prediction/action.
Correlated inputs can make local attributions unstable. Perturbations may create impossible cases. An explanation should be tested for fidelity, stability and usefulness to its intended reader.
For high-stakes decisions, Rudin’s interpretable-model argument asks whether an intrinsically understandable model can meet performance needs before explaining an avoidable black box. Longo et al.’s XAI 2.0 agenda broadens the problem toward actionable, interdisciplinary and falsifiable explanations.
Drift taxonomy
| Drift | What changed | HarborMart signal |
|---|---|---|
| data/covariate | more express orders | |
| prior | lateness rises during disruption | |
| concept | utilisation no longer maps to delay after automation | |
| label | outcome measurement | new delivery-event rule |
| policy | who receives action | threshold or capacity changed |
| feedback | action changes future data | re-promised orders disappear from late labels |
Feature drift alone does not prove performance drift, and stable averages can hide concept failure. Monitor inputs, labels, calibration, decisions and outcomes.
Monitoring card
| Layer | Metric | Trigger | Owner response |
|---|---|---|---|
| data | missing route-distance rate | above 2% for two hours | fall back to rule; inspect feed |
| model | weekly calibration error | outside validated band | recalibrate or investigate shift |
| decision | override rate and reason | abrupt store-specific change | review policy usability |
| outcome | late cost per accepted order | worse than baseline guardrail | rollback |
| fairness | false-negative gap by reviewed group | exceeds approved tolerance | impact review, not silent retuning |
Every trigger needs a window, comparator, owner and action. “Monitor accuracy” is not an operating control.
Airbnb’s Chronon is a current organisational example of treating temporally correct backfills, online/offline consistency and feature monitoring as infrastructure concerns. It illustrates architecture, not a universal vendor prescription.
A compact model card
- intended decision, population and non-use cases;
- training/evaluation cutoff and feature availability;
- baseline and performance with uncertainty;
- calibration and subgroup results;
- threshold, capacity and human override;
- known failure modes and stress tests;
- monitoring, rollback and accountable owners;
- current version and change history.
Quick check
Input distributions are stable, but the same score band now has twice the observed late rate. What drift occurred?