By David Roher
Coach evaluation is a difficult task to approach analytically. I’m currently working on a paper on NBA coaching with other HSAC members, and it’s been very difficult work. Coaches can’t be held entirely responsible for the action on the field, but they exert that limited influence through many different channels. It’s a situation where you have to find a quantity of indeterminate size in an indeterminate number of places. Not fun.
But that doesn’t mean we should forget it altogether. Over the past week and a half, I’ve been developing a simple, intuitive coach evaluation system for the NCAA Football Bowl Subdivision (formerly Division I-A). It’s called RAP, which stands for Resource-Adjusted Performance. Learn about the methodology and the results, after the jump.
The idea behind RAP is that college coaches have a lot more control over their team than their professional counterparts. Not only do they make in-game decisions and playing time calls, but they’re also responsible for the composition of the team through recruiting, and their role in developing athletes is more important than at a professional level. In turn, they receive a lot more credit or blame for the team’s performance.
However, coaches can’t control the resources that are given to them. Southern Mississippi’s football expenses are less than 1/10th of Ohio State’s, and even if they spent the same, Southern Miss still wouldn’t have the same prestige to make recruiting star prospects a lot easier. With that in mind, RAP is essentially the following formula:
(Team Performance Points) + (Team Resource Points)
RAP isn’t a statistic so much as a framework: there are a lot ways to represent team performance and team resource, so it’d be dumb if I didn’t let you choose your own. However, for the sake of comparison, there are a few necessary components:
- The points have to be based on ranking: The worst-performing team gets 1 point, and so on until the best team gets n points, where n is the total number of teams. Similarly, the team with the most resources available gets 1 points, and so on until the team with the least gets n points.
- Team resource points have to be scaled down. Once resource point total is calculated, it has to be multiplied by the coefficient of determination, R2 (which will be between 0-1), between performance rank and resource rank.
- Finally, the sum of the two components has to be scaled so that the average RAP is 1.
The cool thing about RAP is that coaches can be compared across any level of competition or even any sport. There’s no reason you couldn’t separately calculate RAP for FCS (Division I-AA) or Division I Basketball, and the scale would be the same. If resources are more of a factor in some sports than in others, the adjustment for determination would account for that.
The way in which I computed RAP for the FBS was by using the Fremeau Efficiency Index for team performance points, and the 2008 fiscal year expenses of each program for team resource points. If you haven’t heard of FEI, check it out: unlike other computer NCAA rankings, FEI analyzes every single possession, and not only accounts for strength of schedule, but weights those possessions for importance. I’m convinced it’s better than any of the computer rankings that go into the BCS. 2008 expenses come from this fantastic site from the Department of Education, which includes data for all college sports programs except those of the Armed Forces, which means that I had to exclude Army, Navy, and Air Force. It unfortunately does not have data up past 2008 yet.
The R2 between FEI Rank and 2008 Expense rank was .45. So my specific formula was:
(2009 FEI Points) + (2008 Expense Points)*.45
where 85.55 was the average. Here are the top ten coaches by RAP:
|Gary Patterson||Texas Christian||1.69|
|Chris Petersen||Boise State||1.61|
|Paul Johnson||Georgia Tech||1.50|
|Butch Jones||Central Michigan||1.49|
|Mike Riley||Oregon State||1.47|
And the bottom 10:
|Paul Wulff||Washington State||0.43|
|DeWayne Walker||New Mexico State||0.47|
|Mike Locksley||New Mexico||0.56|
|Brady Hoke||San Diego State||0.56|
|Mike Price||Texas-El Paso||0.57|
|Dick Tomey||San Jose State||0.60|
The top 10 coaches all had teams that performed well this year, but to varying degrees: Butch Jones and Houston Nutt coached teams that didn’t even finish in the top 25. And the budgets obviously varied quite a bit as well, although no one in the top 10 in budget appeared in the top 10 in RAP (Mack Brown and the Longhorns were the first of those to appear, at 16th).
RAP is not intended to be perfect, and it certainly is not. There is likely a lot of noise that I’m attributing to the coach’s skill (the previous coach’s prowess, luck, freak injuries, etc.), which is unfair. The calculations I did were based only on 2009 performance, and would likely be more accurate with a larger sample. Additionally, some coaches can affect the resources at their disposal through success and longevity (Joe Paterno and Bobby Bowden being the best examples.) The two statistics I chose to compile it in this case are also problematic: FEI is a proprietary stat, and it’d be nice to have an open-source number for a lot of reasons. And the expense figures are already a bit out of date. The list still goes on.
I want RAP to add to the conversation, not to replace it. By having a number that represents coaching skill, we can do some studies and calculations a little bit better and easier. The first one will be on Monday. Stay tuned.