When Can You Trust Economics Papers?

Felix asks a question dear to my heart. To paraphrase: As a blogger, reporter, columnist, etc., how can you know the non-peer-reviewed paper you're writing about is a good one and not a piece of junk science?

If you have the time, you get a couple of economists knowledgeable in the field to take a look and provide feedback. But for bloggers, who wind up doing the most writing about and linking to working papers, that's not all that feasible. Further, the expertise of Economists isn't without its limits. The criticized paper Felix cites which links hepatits B to "missing" women in China was actually published in the high-flying Journal of Political Economy in 2005. This year, the paper's author, Emily Oster of the University of Chicago, disproved her own findings in an as yet unpublished working paper.

And that isn't the only recent controversy surrounding the Journal of Political Economy, which until recently had economic star Steve Levitt as an editor. Speaking of Levitt, Felix also points to Daniel Davies' episodic critique of Freakonomics as an example of good economic paper criticism, but Davies writes in the first part of his review that he didn't read any of the research backing up the book. That doesn't seem like strong grounds for questioning a researcher's findings, which Davies says he's not doing (UPDATE: Davies writes in the comments that he did in fact read the research):

I'll note at this stage that I have not read the underlying research papers on which Freakonomics is based; normally I would regard this as necessary, but in this case I don't think I should have to bother. I'm not, for the most part, reviewing his academic work or trying to establish the truth or falsity of the claims he makes; I'm trying to establish a particular point about the book Freakonomics, which is that it's not a good book.

I wouldn't disagree on the last point. (To be fair, by the latest installment of the series, Davies says he has read some of Levitt's papers and is even somewhat supportive of the research: "it looks to me as if Levitt is using quite strong instruments.")

Back in 2006 when I was with Reuters, I inadvertently created a mini-uproar by putting out a story about an NBER working paper on the wire with the headline "Taller People Are Smarter." A couple of months later, I flipped open a new issue of the New Yorker to find a Talk of the Town piece on the hate mail the authors of the study, Anne Case and Christina Paxson of Princeton, had received, largely from short men. Who was to blame for this?

The Reuters article ran at three hundred and sixty-three words (the study was fifty-one pages) and simplified the connection that the economists had drawn between height and intelligence.

Which I found funny, since the the researchers wrote this in the abstract:

On average, taller people earn more because they are smarter

And what was glossed over in the Reuters piece? From the New Yorker:

Paxson and Case, who specialize in children's health, theorize that if children experience optimal health and nutrition they will fulfill the maximum height and intelligence of their genetic destiny.

What I had written in the Reuters piece:

Prenatal care and the time between birth and the age of 3 are critical periods for determining future cognitive ability and height.

I'm obviously biased, but I don't know if that qualifies as an oversimplification. Looking back now, though, I could have done a better job of spelling it out. The paper, by the way, is being published in the also high-flying American Economic Review. But the story doesn't end there. Angus Deaton of Princeton -- who is married to Case -- used the incident to write in his biannual Letters From America about the state of peer review in economics (Warning: lenghty quote):

But what does 'published' mean exactly, for a paper that has already been downloaded thousands of times, whose summarized contents have been read by many more thousands, and when the entry case+paxson+height+princeton returns 15,900 hits on Google? Whatever the economics journals are doing, 'publishing' is hardly an accurate description. It has long been the case in the medical sciences that not only the public, but also many professionals, gain their new knowledge from often ill-informed and always incomplete press reports, but medical papers have been (at least minimally) reviewed prior to press exposure. Indeed medical (and most science) journals will not publish papers that have been previously circulated or posted on the web. Perhaps economics journals need to move in this direction, promising rapid review in exchange for eliminating unreviewed dissemination....as this story makes clear once again, the most important 'journal' in economics is the NBER working paper series. It provides a window through which the world watches economics, and where new work that catches the public imagination can be given wide coverage. The National Institutes of Health, which currently funds a great deal of economic research, including Case and Paxson's study, has officially classified the NBER papers, which are not refereed prior to 'publication', as publications for purposes of monitoring old grants and obtaining new ones. For the doctors and scientists who dominate NIH, the idea that publication might take years is too bizarre to credit, and they are relieved to know that the way economists 'really' publish is through the NBER.

Which was then followed by this response from the FT's Tim Harford, from which I plucked the following excerpt (Warning: another lenghty quote):

I write the sort of popular media reports that provoke this sort of hate mail, and I should know, because my sense is that the hate mail usually comes directly to me. I am used to it, but it still hurts. Case and Paxson have my sympathy.

Nevertheless I want to argue that academic economists should keep trying to win media coverage of their work. That work is important -- or should be -- and deserves to be widely appreciated.

We economists should be the first to spot a biased sample when we see one: people who send hate mail are outliers. I have realised that I could save myself some heartache if I asked Slate and the Financial Times to tone down their headlines, since the abusers rarely read my columns. But I don't, because I know that for every idiot who fires off a silly email to me, there are a thousand readers who read my words because the headline grabbed their attention.

For example, Angus complains that the Washington Post excised the words 'on average' from the quotation, 'Taller people are smarter'. But I can see why they did: because 'on average' is implicit in most statements in the social sciences and it is boring. To the extent that the redundant qualifier would have deflected any abuse it would also have deflected interested, thoughtful readers. And after all, Case and Paxson's paper does say that taller people are smarter. What we all want is for readers to look a little deeper and find out why.

Harford then goes on to say that the demand for economics research has brought forth a supply of writers like Tyler Cowen, James Surowiecki, David Warsh, and Joel Waldfogel that are more than capable of writing about it:

So let me turn the question around. Here is a capable crowd of writers with a large and thoughtful audience. What can academic economists do to take advantage of this resource?

Angus points in one direction: learn from the way the NBER 'publishes' economic research. Case and Paxson achieved notoriety through an NBER paper, and NBER is the key channel through which academics reach the media. Journalists are in a hurry. In the past fortnight I have been scooped on NBER papers by my own colleagues Steven Landsburg, Joel Waldfogel (twice) and The Economist. I cannot wait for the [American Economic Review].

The journals, as Angus rightly says, need to do better. NBER is fast, presents user-friendly summaries of hundreds of pages of research, and every journalist receives the email alert at the same time. It is also free to journalists who ask politely.

The Journal of Political Economy, the Quarterly Journal of Economics and the AEA journals all refuse requests for free access -- if they bother to acknowledge them at all. For a professional economics columnist it is worth the expense to subscribe to the AEA, but not the entire library...There have to be limits, and for a reporter with a wide beat, those limits are tighter than you think.

But journals will always be on the slow side. Academic departments could easily use off-the-shelf web-based services to alert any interested party to working papers as they become available. Some entrepreneurial economists -- too few -- email me to alert me to interesting work. More than once they have been rewarded with an audience of several hundred thousand readers -- including a few spiteful emailers, no doubt. I hope they thought it was worth it.

Balancing the interests of the journalist and the researcher without hurting the reader is no small dilemma. What the latter party wants, or in an ideal world should get, is accurate information presented in digestable form. In a world where working papers are freely available, what should a journalist with restricted time do? Released everyday are other reports from companies, foundations, governments, and multinational bodies which are on some levels equally as complex and unvetted as scientific research to the beat journalist. Is there a strong case for trusting these sources more? My instinct says no, but I don't have any evidence to back it up.

But if bad working paper reporting has truly become a problem, then it's time for economists to do as Deaton suggests and put barriers to access until a paper is peer reviewed. And it's not out of the question that this could happen: Universities may become more interested in an economist's star-power than how his/her research holds up. The former can be a lot easier to build up than the latter to tear down.

But as long as policy decisions are made based on little published evidence, as with New York City's pay for grades program, and as more and more research on the subprime debacle comes out -- research that will likely influence decisions in DC long before seeing peer-review --  it doesn't make sense not keep a close -- and always critical -- eye on new research.