What's Right With FICO

Aaron Pressman of BusinessWeek wrote this criticism of FICO in an earlier post:

The increasing reliance on FICO, as a way to compensate for other risk factors going up, was fatally flawed...FICO score became increasingly easy to game. FICO score became less reliable measure of future payment likelihood.

Pressman does a nice job detailing how FICO could be gamed in this February story. And even CIBC star analyst Meredith Whitney said earlier this year that FICO scores are "virtually meaningless." 

Yet the question remains: Was gaming so widespread that it killed the trustworthiness of FICO as a measure of someone's ability to pay?

Let's look at the evidence. (Pulled from Table 12 of this paper -- thanks to Rajdeep Sengupta, one of the authors, for pointing it out to me.)

If FICO was no longer working, then we'd expect to see that subprime borrowers with FICO scores below 620 might actually have a lower chance of defaulting than borrowers with FICOs above 620. Why 620? That was the traditional rule-of-thumb used by the lending industry for deciding who was subprime.

The following four charts show how different subprime vintages performed after origination.

The key points:

- Even though default rates went up, they went up for everybody.

- FICO scores performed just as well in 1998 -- way before subprime securitization -- as they have in recent years.

One caveat is that the difference between the default rates after 4 years for the two lowest credit score groups declined by half between the 2004 and 2005 vintages. That could be a sign that more gamed FICO scores creeped up above the 620 level, but it's more likely a reflection of the impact of falling home prices.

P.S. I originally made four charts showing the evolution of default rates by FICO, but thought the presentation above was clearer. But for those who are interested, I've pasted them below: