When Academic Papers Aren't Replicable

Falsifiability and replicability are key cornerstones of any academic research. If you're running an empirical study, and your results aren't replicable, your study is largely worthless.

But it turns out that nearly all empirical research is worthless on this criterion, according to a devatating new paper by B. D. McCullough and Ross McKitrick. In economics especially, if you try to replicate an empirical result, you'll probably find that you can't get the data -- and if you can get the data, you'll still be a long way from true replication:

Few journals require that data be archived, and those that have such requirements do not reliably enforce them. Even fewer require authors to disclose the software they used for statistical computations. As a result, the time cost of attempting to replicate published studies is extremely high, and replication efforts are rare. The few systematic attempts of which we are aware give surprisingly strong grounds for pessimism regarding the veracity and reproducibility of much empirical research...

In practice it is rare for scientists to make their data and code accessible, and it is rare for scientists to replicate one another's work, in part because it can be so difficult to get the data.

In one study, researchers whose papers had appeared in the Journal of Money, Credit and Banking were asked for their data and code (which explains in detail what is actually done to the data). Most didn't provide it, not that providing it was a huge help:

Failure to replicate was commonplace, even with the active assistance of the original author.

This is a known issue:

To the limited extent the question has been researched, the evidence strongly suggests that most results published in economics journals are not independently reproducible, or verifiable. Where verification is feasible in principle, the time cost to do so is typically very high, requiring multiple requests to reluctant or hostile authors, resulting in underprovision of replication studies.

Science has had woefully little in the way of replication studies for decades: most self-respecting scientists have precious little interest in pure replication work, and given the difficulty in doing it, it's easy to see why it doesn't have it. Still, this is quite a startling datapoint:

Labour Economics called for replication papers in 1993, but dropped the section after 1997 because they had received no submissions.

The paper has a lot of examples of papers which had problems being replicated; it's easy to get bogged down in individual merits of each case, and I'm deliberately not even mentioning any of them by name. The important point is the bigger one: scientists are being far too selfish with their data, and most empirical papers aren't replicable in any kind of realistic sense. This is a big problem, especially given that the peer-review process generally makes no attempt to check on the quality of data or the computations that are done with it.

Interestingly, no one at the major academic journals seems to think that this is much of a problem: there are numerous cases of a policy being put in place requiring authors to submit their data and code, only for the policy to be ignored, or the data to subsequently disappear.

I'm not sure how it's possible to get economists in particular, or academics generally, to spend more time trying to replicate prior results. But it's clearly a major weakness in the academy, and it would be great to address it somehow.