Reproducible Analytics Workflow
Reproducible Analytics Workflow
Reproducibility means more than rerunning a notebook. Another analyst must be able to reconstruct the data available then, execute the transformation and explain why the resulting number supports the decision.
The analysis manifest
| Item | HarborMart example |
|---|---|
| question version | delivery-promise-v3 |
| decision cutoff | checkout confirmation timestamp |
| source snapshot | warehouse tables as of 2026-06-30 23:59 UTC |
| query or extract hash | immutable identifier for extraction logic |
| data dictionary | field, unit, allowed values, owner |
| code environment | Python/package versions or browser runtime |
| random seed | fixed where randomness is used |
| split definition | weeks 1–18 train, 19–21 validation, 22–24 test |
| metric definitions | formulas, denominator and exclusions |
| model/policy version | artefact plus threshold and constraints |
| reviewer | person independent of original calculation |
Saving only the final CSV loses the query, cutoff and exclusions. Saving only code loses the historical source state.
Separate work by purpose
The decision record should contain both the model output and the action actually taken. Otherwise overrides disappear and later evaluation confuses model performance with policy performance.
Tests at four layers
| Layer | Example test |
|---|---|
| schema | order_id non-null and unique at order grain |
| semantic | promised end is after promised start |
| statistical | daily order count within investigated operating range |
| decision | no feature has availability time after checkout |
A passing schema does not establish a sensible business value. A statistical anomaly is not automatically an error; it may be the event the business needs to understand.
Reproduce, then verify
These are different:
- reproduction: rerun the same code and obtain the same result;
- verification: independently check whether the result answers the intended question;
- validation: test performance on appropriate unseen or future cases;
- audit: inspect controls, evidence, responsibility and change history.
Two analysts can reproduce the same leaked model. Reproducibility is necessary, not sufficient.
Responsible use of generative AI
Record where AI assisted with code, text or classification. For each material output:
- preserve the input context without exposing confidential data;
- verify functions and claims against authoritative documentation or source data;
- test edge cases, units and temporal logic;
- identify the human accountable for acceptance;
- keep a non-AI route for sensitive or prohibited data.
An AI-generated query that executes successfully can still multiply rows, use a future field or silently change a denominator.
Reproducibility handoff
A compact handoff contains:
README: question, action, owner and run order;- data contract and source snapshot identifier;
- executable transformations and tests;
- frozen evaluation set definition;
- result table with uncertainty;
- policy logic, override and rollback rules;
- model/data/decision monitoring owners.
Quick check
The model file and notebook are versioned, but the classification threshold is edited manually in a dashboard. Is the decision reproducible?
Answer
Production perspective
Hidden Technical Debt in Machine Learning Systems explains why data dependencies, feedback and undeclared consumers create system risk. Google’s ML Test Score turns this concern into concrete pre-production and monitoring tests.