More on N.C.A.A. Point Shaving: Justin Wolfers Responds

I asked Phil Birnbaum at the Sabremetric Research blog to look at this study claiming to refute research by UPenn economist Justin Wolfers that found signs of point shaving in the N.C.A.A. (Birnbaum is also the editor of By the Numbers, a newsletter which covers the statistical side of baseball.)

The paper (by Dan Bernhardt of the University of Illinois at Urbana-Champaign and Steven Heston of the University of Maryland) exploited the fact that "winning margins for strong favorites are far more likely to fall just short of the spread than to just exceed the spread."

Here's a graph of this effect from their paper:

Under Wolfers' view, this statistical anomaly could represent point shaving: Gamblers paying players or officials to change the natural flow of the game in such a way that the final spread is just less than the Las Vegas line.

For his part, Birnbaum loved the paper:

They took a statistical effect that Wolfers claimed showed corruption, and proved, four different ways, that the effect exists even when corruption is unlikely. In addition, they provided a plausible explanation of what the effect might be.

The NCAA should buy these guys a beer.

Not quite, says Wolfers. (Both in a lucid comment on Birnbaum's blog post and in a phone conversation with me.)

While he likes the new study, the problem lies in the way Bernhardt and Heston sliced up the numbers, Wolfers says.

The researchers divided the sample in to two parts:

• One set included those games where the opening spread was wider than the closing spread, meaning more money was bet on the underdog than the favorite. Bernhardt and Heston say that under these circumstances it's plausible that gamblers paid players on the favorite to play badly in order for their team to win by less than the spread.
• The other set included games where the closing spread was wider than the opening spread, meaning more money was bet on the favorite. The researchers claim that this means there could have been no point shaving in these games.

Bernhardt and Heston found that the statistical anomaly existed 6.64 percent of the time in games where the spread was tighter at the end and 5.61 percent of the time in games where the spread was wider -- both of these numbers are very close to Wolfers' results. Since the results from Bernhardt and Heston's two different samples are so similar, this led them to conclude that the statistical anomaly was just that, not something illicit. If anything, the difference between the two numbers, 1.03 percent, might be the true level of point shaving. (Though the pair say that this result is not statistically different from zero.)

But there's a problem with their test, says Wolfers.

In preparation for his paper, he had an M.B.A. student track down as many historical cases of point shaving in U.S. sports history as possible -- roughly 40. While he calls it "low-level journalism rather than real research," Wolfers says about 1/3 to 2/3 of the cases didn't show signs of point-shaving after-the-fact. This means that what Bernhardt and Heston consider a pure sample could actually be tainted with cases of point-shaving.

Wolfers:

"In fact, if you look at the history of known point-shaving episodes over the past century, you will often find that the betting line didn't move. There are two reasons for this. Some point shaving episodes are just half-assed - a college kid betting a few hundred dollars on the opposition, and this doesn't move the betting line. And second, only a very very small proportion of gambling on NCAA basketball occurs through Las Vegas, the source of the betting line movements relied on by the authors."

(An interesting test case is the N.B.A.'s current point shaving scandal: How much did the Las Vegas point spread move in games officiated by ref Tim Donaghy?)

Bernhardt and Heston also construct another control sample -- N.C.A.A. games in which there was no point spread, meaning point-shaving was impossible. For this set of games the pair created their own spreads based on team strength. They found that the data suggested "point shaving" in these games occurred 5.91 percent of the time.

But Wolfers has a problem with this test too. He believes the 5.91 percent result represents the researchers own error in trying to mimic point spreads.

"Betting markets are a lot smarter than econometric models produced by even very clever economists, and so I suspect that some of this result reflects them mis-predicting what the spread would have been."

Wolfers says his original paper is more robust because his sample size (about 44,000 games) was much larger than the one used by Bernhardt and Heston (about 9,000 games for the first test).

In the end, because they do find similar results, Wolfers argues that the new paper strengthens his original one.

"If you take enough cuts of the data in enough different ways -- and take enough of it out -- you can't falsify anything," Wolfers said. "This is absence of evidence versus evidence of absence."