Data Contracts
Data Contracts
One row must mean one thing
HarborMart stores three different grains:
| Table | One row | Candidate key |
|---|---|---|
orders | one order | order_id |
order_lines | one product line within an order | order_id, product_id |
delivery_events | one status event | order_id, event_time, event_type |
Joining orders to order_lines multiplies order rows. Summing order_value after that join double-counts orders with several lines.
Small example
| order | order value | lines |
|---|---|---|
| A | £30 | 3 |
| B | £20 | 1 |
The true total is £50. After a line-level join, summing repeated order values gives . The arithmetic is correct; the grain is wrong.
The five-clock audit
For every field, record:
| Clock | Meaning | Example |
|---|---|---|
| event time | when the world changed | order placed 17:03 |
| observed time | when the system captured it | payment event 17:03:04 |
| available time | when a decision service could use it | feature store 17:04 |
| decision time | when action was selected | promise confirmed 17:03:10 |
| label time | when the outcome became knowable | delivery completed 19:42 |
A feature is valid only if its available time is no later than decision time. A final route duration entered after delivery is not a predictive feature for checkout, even if its event date is the same day.
A minimum data contract
| Field | Required statement |
|---|---|
| population | all eligible UK delivery orders accepted in completed weeks |
| exclusions | test accounts and documented outages; cancellations retained as a separate outcome |
| unit | one accepted order |
| key | stable unique order_id |
| target | delivered after promised end timestamp |
| feature cutoff | checkout confirmation time |
| timezone | Europe/London with daylight-saving handling |
| currency | GBP gross or net of VAT, stated consistently |
| missingness | valid absence, delayed record or system failure distinguished |
| revision policy | which late-arriving events restate historical metrics |
Missing is a process observation
Do not begin with mean imputation. Ask why the value is absent:
- structural: apartment floor does not apply to a house;
- random capture failure: scanner briefly offline;
- selective: difficult deliveries omit a completion code more often;
- not yet available: outcome still developing.
An indicator plus domain review may be more honest than replacing every blank with an average. If missingness depends on the unobserved value, routine imputation can preserve bias while making the table look complete.
Join checks that must balance
Before and after each join, record:
- row count and distinct key count;
- unmatched keys on each side;
- duplication factor by key;
- totals that should be invariant;
- distribution of newly missing fields.
For the £50 example, sum(order_value) is an invariant only at order grain. At line grain, allocate value or aggregate lines before joining.
Data lineage in one sentence
late_flagis computed nightly from immutable promise timestamps and the first valid delivered event, with source-system version, extraction time and transformation code recorded.
This sentence identifies source, rule, timing and reproducibility. “From the warehouse” does not.
Quick check
Why is a customer’s current lifetime order count dangerous in a historical churn-training row?
Answer
Further evidence
The dataset-documentation literature treats provenance and intended use as part of the data object, not optional prose. Start with Datasheets for Datasets and adapt its questions to the decision context.