Distributions and Exploratory Data Analysis
Distributions and Exploratory Data Analysis
The average is one coordinate
HarborMart observes delivery delays, in minutes relative to the promised end time:
The mean is 4.7 minutes late, while the median is 2.0 minutes late. Neither is wrong:
- the median describes a typical order;
- the mean includes the operational burden of the 48-minute failure;
- the upper tail matters for refunds and trust.
Report the statistic that matches the decision, but show enough of the distribution to reveal what it omits.
A five-question profile
| Question | Useful summary | HarborMart interpretation |
|---|---|---|
| where is the centre? | median and mean | typical versus total burden |
| how variable is it? | IQR, standard deviation | routine dispersion |
| how bad is the tail? | 90th/95th percentile, exceedance rate | service-risk exposure |
| is it multimodal? | histogram or grouped quantiles | perhaps two fulfilment processes |
| does it differ by context? | stratified distributions | store, slot, basket or weather |
The interquartile range is . It resists a few extreme values; that robustness is useful, but it can also make important failures disappear from the headline.
Outlier is not a deletion instruction
For the 48-minute delay, ask in order:
- invalid record? impossible timestamp or duplicate event;
- valid exceptional event? vehicle breakdown or road closure;
- valid recurring process? a particular store or time slot;
- decision relevance? costly enough to model separately;
- sensitivity? does the recommendation change with and without it?
Deleting every point beyond changes a descriptive convention into an undocumented business rule.
Audit centre, spread and tail
Change 48 to 480. Notice which summaries move and which stay relatively stable. The correct response is not automatically to prefer the stable statistic; it is to identify whether the event belongs to the target process.
EDA is structured scepticism
Use this order:
- verify units, keys, dates and impossible values;
- inspect univariate distributions before relationships;
- compare missing versus observed groups;
- stratify by variables known before the outcome;
- inspect time order and operational changes;
- write hypotheses separately from confirmed findings.
Exploration generates questions. Repeatedly searching until a pleasing pattern appears does not turn that pattern into confirmatory evidence.
Quick check
Store A has a lower median delay but a much higher 95th percentile than Store B. Which is “better”?