Reproducibility and Reporting
Reproducibility and Reporting
Reproduction has levels
| Goal | Question |
|---|---|
| computational reproducibility | does the same code and data regenerate the result? |
| robustness | does the conclusion survive defensible analytical changes? |
| replication | does new data or an independent implementation support the finding? |
Passing the first is necessary but does not establish the other two.
The one-command principle
A reviewer should be able to start from documented inputs and run one orchestration command that:
- validates expected files and versions;
- builds analytic data;
- estimates all reported models;
- generates tables and figures;
- runs tests and writes a machine-readable log.
This repository already uses a locked JavaScript dependency workflow and VitePress build. Course cells are enhancements: every substantive result also appears statically, so learning and review do not depend on a browser runtime.
Minimum replication package
| Component | Evidence |
|---|---|
| README | sequence, expected runtime, hardware and output map |
| data manifest | source, version, licence/access and checksums |
| environment | lockfile, language and package versions |
| code | numbered or dependency-driven scripts; no manual steps |
| seeds | every random split, simulation and bootstrap controlled |
| tests | keys, row counts, domains and known benchmark estimates |
| disclosure log | deviations, exclusions, failed analyses and restrictions |
| outputs | generated from code; figures include underlying values |
Absolute local paths, undocumented spreadsheet edits and “run cells in this order until it works” are reproducibility failures.
Restricted data are a documentation problem, not an exemption
The AEA Data and Code Availability Policy, in its February 2026 version, expects enough data, code and computational detail to permit replication while providing procedures for legally or ethically restricted data. A package can therefore include access instructions, executable code, synthetic fixtures and a controlled-computing description without publishing protected records.
The policy is a journal requirement, not a universal rule for every course project. Use it here as a high-quality audit benchmark and check the live requirements of the intended venue.
Make randomness and computing visible
Record:
- pseudorandom generator and seed;
- fold/cluster construction;
- bootstrap or permutation repetitions;
- CPU/GPU architecture when numerically material;
- solver tolerances and convergence status;
- package and system-library versions;
- elapsed time and expected memory.
For parallel procedures, a single top-level seed may not reproduce task scheduling. Test the package on a clean environment rather than the author’s long-lived machine.
A results contract
Add tests around substantive invariants:
- exactly one row per applicant in the cross-section;
- offer assignment precedes every outcome date;
- lottery share lies in its documented tolerance;
- the primary ITT uses the declared denominator;
- all reported table values are generated objects;
- no unreleased or direct identifiers enter outputs.
Tests do not guarantee truth. They make silent changes observable.
Quick check
The analysis runs only because a package was installed globally on the author’s laptop. Is a list of package names sufficient?