When Can You Trust Economics Papers?

I had a great lunch with Columbia's Ray Fisman a couple of weeks ago, and I've been meaning since then to blog about one of the big problems with what you might call Freakonomics-style economics, where economists sift through statistical data and come up with some kind of interesting insight. That's something Ray does quite a lot of himself, and he and Ted Miguel run through quite a lot of such studies in their new book, Economic Gangsters, which applies such thinking to questions of development.

But one of the problems with such research is the way that it's presented to the public, through newspaper and magazine articles, or TED lectures, or the online dissemination of pre-pubication versions of papers. The stuff which gets the most attention has very rarely gone through the peer-review process, and as a result it's often impossible for us, the final consumers, to get a good grip on what's worth taking seriously and what's just junk science. In August, Stephan Klasen called out Robert Barro of Business Week, as well as Steve Levitt himself, in Slate, for writing glowingly and uncritically about a paper of Emily Oster's, which she has since renounced, before it had gone through any kind of peer review.

Now that Fisman himself has Levitt's old perch at Slate, I asked him about this issue, and about the phenomenon whereby even as papers get a lot of play in the blogosphere, economists are generally still reluctant to make substantive criticisms on public blogs rather than in private seminars. (This doesn't apply to policy debates, of course: they're a different kettle of fish entirely.) The result is that economic papers get much more pre-publication attention than ever before, with most of it being very uncritical. By the time the paper's gone through peer review, public attention has moved elsewhere.

It helps of course that Fisman is an economist himself, and therefore is reasonably well qualified to judge the papers he's writing about. He wrote to me:

There are lots of things that are oh so very interesting that I'd love to put up in Slate. But unfortunately when I actually read the studies, I don't believe they're true. I'd also say that for topics beyond my expertise, I will always consult with people in that particular field - education economists for teacher quality, someone at teachers' college on computers in classrooms.

Journalists do this too -- not that it necessarily helps, because many scientists now come with PR people attached, who will pitch a story to dozens of journalists. If the paper is crap and most of the journalists turn it down, harm is still done, because the few lazy journalists who do pick it up are even more likely to just regurgitate whatever the PR person gives them and not even ask for a copy of the paper in question. From a reader's point of view, it's often impossible to tell how assiduous the journalist has been in writing the story. And even the most careful journalist is likely to miss a lot of the kind of issues which Daniel Davies brings up in his discussion of Freakonomics.

So what to do when a book like Fisman's comes out, where an economist talks about his own work? Is such a book even less reliable, since it's unfiltered by even a journalist? No: the good thing about books is that they don't have the same amount of urgency, and that therefore the research in Economic Gangsters has all been peer-reviewed.

But the fact is that when you read about economic research, unless you're an economist yourself you're extremely unlikely to get a solid grip on how seriously to take it. I briefly tried to act as a referee for such things, but that didn't work out so well: in my very first outing, it was I who was wrong.

So the best one can do, really, is bear in mind a dictum from John Allen Paulos: "Always be smart; seldom be certain." If you think a paper is interesting, fine. If you think it's compelling, then think again: there's a good chance there's something you're missing. I need to learn this lesson as much as anybody: back in May I gushed about a paper linking a falling violent-crime rate to the introduction of unleaded gasoline. I still think the result is true, but I'm more alert, now, to the fact that I'm quite unqualified to make that determination.