Can Ratings Agencies be Fixed?

Stanford's Darrell Duffie is giving a series of lectures at Princeton on capital immobility, which will be turned into a book to be published by Princeton University Press. The Press invited me to dinner with Duffie last night – delicious, thanks! – and we got to talking about ratings agencies. Duffie is a member of Moody's Academic Research Committee, and his take on the ratings crisis is basically that there's nothing there that better modelling and clearer thinking can't fix.

What's the problem, he says, with Moody's rating structured products like the notorious CPDOs? Not that Moody's got the rating wrong, so much as that the product should never have been considered a ratable instrument in the first place. Is there a conflict, he asks, inherent in the fact that issuers, and not investors, pay the ratings agencies? No: the ratings agencies' credibility is much more important to them than any benefit they might get from boosting the ratings they give, and in any case it's relative ratings which matter, not absolute ones.

I wasn't convinced. For one thing, the ratings agencies clearly and consistently said that ratings are horizontally comparable, as it were: that a double-A rating on a sovereign means it has the same default risk as a double-A rated municipal bond, or corporate bond, or structured product. And it turns out that's not the case: structured products, especially, default much more than identically-rated munis, and they always have – this is not news. It certainly seems as though the ratings agencies, which made enormous profits from rating structured products, were "nicer" to those products than they were to other issuers.

But my real reason for skepticism is that I'm in the middle of being extremely impressed by another Princeton University Press author, Riccardo Rebonato. In his new book, Rebonato compellingly skewers the "frequentist" approach to probabilities employed not only by Moody's and the other ratings agencies but more generally across the whole world of finance. Looking backwards at what happened in the past gives you lots of data, which can be chopped up and examined in any number of different ways, gives a false sense that future probabilities can be scientifically determined to three or four significant figures.

But that whole approach is based on the idea that market moves are like coin flips, or balls in an urn: that the markets might move, but that their underlying structure never really changes. In fact, the defaults rates that matter are the ones in the future, not the ones in the past. And to get a grip on those you need to understand what Rebonato calls "subjective probability" – which, although it might not have quite the mathematical rigor of frequentist probability, can actually be much more useful and accurate.

Duffie does understand this. Moody's had a habit, he says, of tweaking incredibly complicated models until they matched the past data, and then deciding that because the models matched the past data, they must give a good idea of what will happen in the future. Obviously, that's never going to work very well: it's the financial equivalent of shooting an arrow at a barn door, painting a target around it, and claiming astonishing marksmanship. But I think Duffie and I differ on the question of whether better models can fix this problem. He thinks they can; I think they can't, certainly so long as Bayesian statistics remains a backwater for Wall Street's quants.