Worked Examples and Pitfalls

1. A correct average can tell the wrong story

Store B is worse within both channels but looks better overall because it serves many more low-risk mobile orders. This is a composition reversal, not a calculation error.

Decision: compare like with like, show weights, then ask whether standardisation or a model is appropriate. Do not choose a store intervention from the pooled rate alone.

2. The denominator can reward avoidance

Of 1,000 accepted orders, 950 are completed and 902 meet the original promise.

• on time among completed: $902/950=94.9\%$;
• original promise met among accepted: $902/1{,}000=90.2\%$;
• cancellation rate: $50/1{,}000=5.0\%$.

If cancellations rise, the first KPI can improve while customers receive worse service.

Decision: use a metric family: completion, original-promise service, revised-promise service and cancellation. Freeze definitions before an experiment.

3. A one-to-many join changes the unit

There are 100 orders and 300 item rows. Joining orders to order_items repeats an order once per item. If larger baskets are more often late, a row-level mean estimates an item-weighted late rate, not an order late rate.

Repair: aggregate item features to one row per order_id, verify row count and key uniqueness, then join. State when an item-weighted question is actually intended.

4. Leakage often looks like excellent prediction

A checkout model includes actual_picker_minutes. Cross-validation split randomly across rows gives AUROC 0.93. But actual minutes are finalised after the delivery shift.

Repair: create an availability table for every feature, reconstruct values as known at checkout, and split by time. A weaker honest score can support a real decision; the stronger leaked score cannot.

5. Accuracy can reward doing nothing

Only 3% of customers churn. Predicting “no churn” for everyone is 97% accurate, with zero recall for churners.

Repair: compare against that baseline; report confusion counts, precision–recall behaviour, calibration and expected value at the actual contact capacity. Do not replace one context-free metric with another.

6. Risk is not treatment response

A risk-ranked campaign chooses A; an effect-and-value policy chooses B. Both effect estimates still need credible causal evidence and uncertainty.

7. A forecast does not determine inventory

Two products have the same mean demand of 20 units. Product F is perishable with high overage cost; Product S is durable with high stockout cost. Their optimal quantities differ because the critical fractile depends on $C_u/(C_u+C_o)$, not only the mean.

Decision: estimate a demand distribution, define salvage, shortage and service consequences, then test the order quantity under alternative costs. “Forecast 20, order 20” is not a policy.

8. An optimiser obeys the wrong model perfectly

A schedule maximises contribution subject to picker hours and returns 140 orders. It omits refrigerated-van capacity, so only 110 can be delivered safely.

Repair: run a feasibility review with operators, audit units, compare solver output with a hand-checked case, and record excluded constraints. If a constraint is difficult to model, it does not cease to exist.

9. A stable model can support a failing policy

Calibration is unchanged, but staff acceptance drops from 80% to 35% after a new rota. Investigation shows recommended shifts now violate contracted availability.

Decision: monitor recommendation feasibility and execution, not only scores. Retraining the model would treat the symptom as if it were a statistical failure.

One-minute audit

Before accepting any result, ask:

1. What is the unit and denominator?
2. Was every input available at the decision time?
3. Is this a descriptive, predictive, causal or prescriptive claim?
4. What baseline and uncertainty accompany it?
5. Which action changes, and what happens if the assumption is wrong?

Next: Reading and Evidence Map