The RPI is Not the Real Predictive Indicator

Despite taking quite the beating recently, the Ratings Percentage Index (RPI) is still the official and favored ranking system of the NCAA Men’s Basketball Selection Committee. The RPI does not take into account margin of victory, making its predictive accuracy dubious at best when compared to rankings like Ken Pomeroy’s or Jeff Sagarin’s, which do include MOV. That is why I was stunned to hear Mike Bobinski, chair of the Selection Committee, say this last week:

“Interestingly, last week we asked a statistician that works with the NCAA who is really, really sharp, to sort of do a comparison of all the major different rankings that exist, including the RPI and others that you can all probably certainly come up with who they are, and compare those evaluations systems with performance in the tournament.

Interestingly, if we went through that, we were all surprised to see that the RPI actually did end up with the highest level of predictive value and the highest correlation with ultimately success in the tournament. That doesn’t mean we’re going to use it more or less this year. It’s just a very interesting piece of information.”

If that were true, it would run counter to all that we think we know about predicting college basketball games and assessing past performance. As Ken Pomeroy has shown recently, scoring margin in past matchups matters  for predicting future matchups between two teams. I don’t doubt the sharpness of the NCAA’s statisticians, but I decided to do the analysis for myself.

As the results show, the claim that the RPI is the best predictor of NCAA Tournament results is absolutely wrong.

To evaluate the NCAA’s claim, I took the final RPI rankings prior to the NCAA Tournament for each year from 2007 through 2012 from Statsheet and RealTimeRPI and compared them to Ken Pomeroy’s Pythagorean ratings from just prior to the NCAA Tournament and my own Survival Analysis model, which incorporates Ken’s Adjusted Offensive and Defensive Ratings.

For each season, I filled out three brackets, one per system, by advancing the higher-ranked team in each game. While a matchup between two teams with very similar rankings in each system is largely equivalent to a coin flip, making simply advancing the higher ranked team not optimal, basketball games result in binary outcomes and any predictive system has to be evaluated in that framework. I looked at two primary measures: games projected correctly and the bracket score using the standard 1-2-4-8-16-32 point scoring system employed by ESPN.com and Yahoo’s bracket pools.

It matters not only how many games a system predicts, but also which games it predicts correctly. It is more valuable to correctly identify the NCAA Champion or Final Four teams than the winner of a first round game, and the prediction systems should be judged accordingly.

The table above summarizes the results of predicting the 2007-2012 NCAA Tournaments. The RPI is close to Ken Pomeroy’s rankings in terms of average games predicted, but is blown away by the Survival Model. Additionally, the RPI is much worse on the dimensions that matter: predicting Final Four teams and NCAA Champions. Both the Survival Model and Pomeroy’s rankings score far better and have identified more championship teams before March Madness begins.

The 2008 NCAA Tournament is instructive as to why the RPI is not as predictive of future results. Prior to the Tournament, the RPI ranked Tennessee as the best team in the country. While the Volunteers had had a tremendous season, they had also relied heavily on narrow wins. The Vols were 9-1 in games decided by fewer than five points to that point in 2008, and the SEC was one of the weakest major conferences.

Kansas, by contrast, had a much stronger margin of victory while running through the Big 12, the best conference in the country. Both Pomeroy’s model and my model ranked the Jayhawks first before the Tournament began. Pomeroy had the Vols ranked 19th; I had them ranked 23rd. As we now know, Kansas cut down the nets in a classic title game, while Tennessee lost in the Elite Eight to North Carolina. Ignoring margin of victory, as the RPI does, throws away valuable predictive information.

If you had filled out your bracket according to the RPI, you would have lost your office pool to a Survival Analysis bracket in every year since 2007, and only beaten a Ken Pomeroy bracket in one year, 2007. Even in 2011, when three-seeded Connecticut won and all predictive models did poorly, the RPI was the worst of the bunch.

The claim that the RPI is the most accurate predictor of NCAA Tournament results is simply not true. The Selection Committee uses the RPI as a main seeding criterion, which makes its relative failure as a predictive system even more glaring. If the brackets were seeded according to Pythagorean Expectation, Ken Pomeroy’s predictions would likely be more accurate.

The NCAA has its own– in my view, misguided– reasons for continuing to use a ratings system that does not account for margin of victory. One of those stated reasons, however, should not and cannot be its predicted accuracy. If you are trying to win your March Madness pool, you wouldn’t fill out your bracket according to the RPI. The NCAA should not set up its bracket according to the RPI, either.