Capstone — The Evening Delivery Promise
Capstone — The Evening Delivery Promise
The decision
HarborMart's regional manager asks: “Can the risk model reduce evening delivery failures?” Your task is not to praise or reject the model. You must design a defensible decision system.
At checkout, the system may:
- accept the requested window;
- offer a later promise;
- reroute to a nearby fulfilment centre when feasible; or
- call additional picker capacity for a future slot.
The customer must see the promise before payment. Information observed after that moment is unavailable to the decision.
Evidence pack A: eight-week operating summary
There were 24,000 completed evening orders and 312 cancellations after acceptance.
| Window | Completed orders | Late | Late rate |
|---|---|---|---|
| 16:00–17:00 | 6,000 | 330 | 5.5% |
| 17:00–18:00 | 7,000 | 560 | 8.0% |
| 18:00–19:00 | 6,500 | 715 | 11.0% |
| 19:00–20:00 | 4,500 | 770 | 17.1% |
| total | 24,000 | 2,375 | 9.9% |
This table does not prove that the time window causes lateness. Store, distance, basket and capacity mix also change through the evening.
Evidence pack B: data dictionary
| Field | Grain or timing | Audit note |
|---|---|---|
orders.order_id | one accepted order | primary key; cancelled orders remain |
order_items | many rows per order | joining before aggregation duplicates orders |
score_at_checkout | stored before promise | legitimate historical prediction |
scheduled_picker_minutes | known at checkout | legitimate capacity feature |
actual_picker_minutes | final shift record | leakage for checkout prediction |
weather_forecast_at_checkout | forecast version stored at checkout | legitimate if timestamp is preserved |
actual_weather | observed later | outcome-time information |
arrival_time | observed for completed delivery | missing for cancellations |
customer_service_note | free text after delivery | possible leakage and sensitive content |
service_area | operational zone | use for error and impact audit |
The late label is arrival_time > promise_end. Decide explicitly how cancellation, re-promise and missing arrival should enter the business outcome; silently dropping them rewards avoidance.
Evidence pack C: model audit
The candidate model was evaluated on the latest four weeks, after all training periods.
| Risk band | Orders | Mean predicted risk | Observed late rate |
|---|---|---|---|
| below 5% | 8,000 | 3.0% | 3.0% |
| 5%–10% | 7,000 | 7.0% | 7.0% |
| 10%–20% | 5,500 | 14.0% | 14.0% |
| above 20% | 3,500 | 25.0% | 25.0% |
| Measure | Constant-risk baseline | Candidate |
|---|---|---|
| Brier score | 0.089 | 0.078 |
| AUROC | 0.500 | 0.720 |
Aggregate calibration hides one issue:
| Service area | Orders | Mean prediction | Observed rate |
|---|---|---|---|
| central and inner | 19,200 | 9.7% | 8.5% |
| outer | 4,800 | 10.6% | 15.4% |
The outer-area result may reflect model error, changed road conditions, weak labels or a different data-generating process. Diagnose before changing a threshold.
Evidence pack D: candidate actions
The finance team values a prevented late delivery at £9 after refunds, support time and estimated retention effects. Treat this figure as an assumption requiring sensitivity analysis.
| Action over eight weeks | Direct cost | Estimated consequence | Evidence quality |
|---|---|---|---|
| add one picker, 17:00–20:00 | £2,160 | 420 fewer late orders, interval 250–580 | randomised shift pilot |
| reroute up to 600 eligible orders | £1,440 | 210 fewer late orders | adjusted observational comparison |
| offer later windows to top 300 risks | £1,800 lost contribution | 90 fewer recorded late orders | model simulation only |
| keep current rule | £0 | baseline | observed history |
Capacity allows only one additional picker. Rerouting is unavailable for chilled baskets and adds a median 11 minutes to travel. A re-promised order is not “on time” under the original promise; retain both original- and revised-promise measures.
Your six decisions
A. Frame
Write an analytics contract naming owner, decision time, unit, eligible population, actions, objective, constraints, baseline, horizon and review rule. State one descriptive, predictive, causal and prescriptive claim the evidence can or cannot support.
B. Audit
Specify the order-level analysis table. Show how you will prevent item-join duplication, temporal leakage and silent cancellation exclusion. Add five executable data checks.
C. Diagnose
Produce one decision display that separates volume from late rate and exposes the outer-area problem. Include counts, denominators and uncertainty. Give two rival process explanations.
D. Predict
Decide whether the candidate improves on the baseline for this decision. Evaluate temporal validity, calibration, ranking, subgroup error and the cost/capacity range. Do not select a threshold from AUROC alone.
E. Prescribe
Recommend an action or staged combination. Calculate expected net value, test the £9 assumption and use the intervention evidence quality. State why “offer later windows to top risks” may optimise the reported metric while harming service.
F. Deploy and learn
Design a shadow stage and a controlled rollout. Specify five monitors, two guardrails, one pause condition, one rollback rule, owners and a causal value evaluation. Explain how selective labels will arise.
Deliverable template
This is a teaching template, not an institutional rule:
- executive decision memo: recommendation, expected value range, material uncertainty and next decision;
- technical appendix: contract, data audit, validation, policy calculation and sensitivity;
- control sheet: monitoring, fairness/privacy review, incident and rollback rules;
- reproducibility record: data snapshot, code/environment, assumptions and generated outputs.
Evaluation rubric
| Criterion | Strong work demonstrates |
|---|---|
| decision logic | objective, alternatives, constraints and counterfactual align |
| evidence discipline | description, prediction, causation and prescription are not conflated |
| technical validity | grain, time, denominator, baseline, calibration and uncertainty are audited |
| operational realism | action is feasible; ownership, latency, feedback and rollback are explicit |
| responsibility | affected groups, privacy, security, review and contestability shape the design |
| communication | conclusion is concise, traceable and honest about what is unknown |
Postgraduate work should add an identification diagram, a formal estimand, robustness to model misspecification and decision-regret comparison.
Instructor checkpoint — not a unique solution
The additional picker has point-estimate gross value and net value £1,620. At the pilot interval endpoints, net value ranges from £90 to £3,060; break-even value per prevented late order is about £5.14. That makes a staged picker rollout defensible, not certain.
Rerouting has point-estimate net value , but weaker causal evidence and eligibility constraints. Re-promising may reduce the recorded late rate by redefining service rather than improving it. Preserve original-promise outcomes.
A strong answer investigates outer-area calibration, keeps cancellations in a guardrail, logs proposed and executed actions, and randomises rollout where feasible. It does not feed actual_picker_minutes, actual_weather or service notes into a checkout-time model.
Use the formula map, then compare your reasoning with the worked pitfalls.
Reading and Evidence Map
A dated, selective map of foundational research, current extensions, standards and organisational cases
Financial and Economic Time Series — Course Guide
A matrix-based course in returns, volatility, persistent predictors, cointegration, structural dynamics, nowcasting, and forecast evaluation.