The Limits of Empiricism

Two interesting reviews today: David Leonhardt on Ian Ayres, and Daniel Davies on Steven Levitt and Roland Fryer. There's a meme catching hold – think Freakonomics and Moneyball – which says that rigorous empirical analysis can reveal otherwise-unobtainable insights. But these reviews, along with Davies's latest Freakonomics review (part four of a long, wonderful, and hilariously delayed series) constitute a rather interesting countermeme.

The problem is that attempts at rigorous empirical analysis have long since eclipsed the number of empirically-rigorous datasets available. And so you end up with Levitt and Fryer running regression analyses on a KKK dataset which is dominated by Pennsylvania and Ohio, or with Ayres gushing about call-center call times on the basis of some gushing coverage in a back-issue of Fast Company.

As Davies points out in his Freakonomics review, insofar as economics works it's largely because a lot of very smart people spent a great deal of time working out how to generate reliable statistics; and also because whenever someone does any kind of economics research, you can be sure that they're using the same statistics as everybody else. Similarly, insofar as Moneyball worked, that was probably due to the fact that baseball is one of the few other places where you can find a set of universally-accepted and exhaustive statistics. But when you start looking at phone-call durations or KKK membership, you've moved a very long way into much squishier territory.