Data Products and Monitoring
Data Products and Monitoring
Treat the model as one component
HarborMart's score is usable only if five interfaces agree:
| Producer gives | Consumer assumes | Contract test |
|---|---|---|
checkout service: order_time in UTC | feature job: event time, not load time | reject future or unparsable timestamps |
| capacity service: available picker-minutes | model: value known at checkout | compare event timestamp with score timestamp |
| model: probability in | policy: calibrated late-delivery risk | range and calibration test |
| policy: accept, reroute or re-promise | operations: action before confirmation | latency and action-log test |
| delivery system: actual arrival | evaluator: final label | label completeness after a fixed delay |
Version the schema, feature definitions, model, policy and cost assumptions together. A score from model v4 interpreted by policy v2 is not reproducible merely because both files exist.
Five monitors, five responses
| Monitor | Example trigger | First response |
|---|---|---|
| service | p95 latency exceeds 300 ms for 10 min | fall back to the tested rule |
| data | scheduled picker-minutes missing above 1% | stop scoring affected stores |
| model | four-week Brier score exceeds the baseline | inspect calibration and segment error |
| decision | override rate doubles | inspect workflow, incentives and incidents |
| outcome | late rate rises while rejection rises | pause rollout; reassess effect and objective |
Choose thresholds from operational tolerance and historical variation. “Alert when a line looks unusual” cannot be tested.
Drift is a diagnosis, not a verdict
- Covariate drift: changes—for example, baskets become larger before a holiday.
- Prior shift: changes—for example, storms raise the late-delivery base rate.
- Concept drift: changes—for example, a routing redesign changes how distance predicts lateness.
- Policy drift: the same score produces different actions because capacity or staff behaviour changes.
A large feature shift may be harmless; a small shift in a high-cost subgroup may matter. Compare distributions, calibration and decision value, then investigate the process that generated the change.
Worked incident
On Monday, HarborMart deploys a faster routing process. By Friday:
| Signal | Before | After |
|---|---|---|
| predicted late risk | 8.1% | 8.0% |
| eventual late rate | 8.0% | 5.6% |
| actions labelled after 48 hours | 99.4% | 71.0% |
Two explanations fit the early dashboard: the intervention helped, or late labels are incomplete. Do not recalibrate until label delay and missingness are resolved. First freeze the evaluation cohort by decision date and wait for its defined maturity window.
Selective labels and feedback
If high-risk orders are rejected, their delivery outcomes are never observed. The deployed policy changes its own future training data. Similarly, extra staff assigned to high-risk slots can make the model appear pessimistic because the intervention prevents the predicted event.
Log at least:
- the information and score available at decision time;
- the proposed and executed action;
- override actor and reason;
- exposure or treatment actually received;
- outcome and its observation time.
Without this decision log, one cannot separate model error from successful intervention.
Release ladder
| Stage | What changes | Exit evidence |
|---|---|---|
| replay | nothing operational | historical pipeline reproduces known results |
| shadow | scores logged, no action changed | latency, coverage and calibration acceptable |
| limited pilot | a small eligible unit receives the policy | safety and outcome stop rules remain clear |
| staged rollout | exposure expands by store or time | stable effects and no material subgroup harm |
| routine operation | policy is default | recurring review, challenger and rollback tested |
Keep a simple fallback that does not depend on the failing component. Test rollback before it is needed.
Current organisational example
Airbnb describes Chronon as an end-to-end feature platform supporting temporally accurate backfills, online/offline consistency and automated monitoring for data quality, training–serving skew and feature drift. The transferable lesson is architectural: a feature is a time-indexed data product with producers, consumers and controls—not a column copied into a notebook.
This is an organisation's engineering account, useful for system design but not independent evidence of business impact.
Quick check
The score distribution and AUROC are stable, but staff now override 45% of recommendations instead of 8%. Is the system healthy?