Commissioner Roger Goodell at the NFL Draft
by Kevin Meers
Through 12 weeks of the season, most franchises in the National Football League know whether they are in the playoff hunt or if they need to start prepping for the draft: The Green Bay Packers and San Francisco 49ers have almost clinched playoff berths while the Indianapolis Colts have just about locked up the number one overall draft pick. As the Superbowl hopefuls get excited for playoff runs, most of the league has to sit and wait for April. For five months, most teams have to sit and wait. They are on the outside looking in, watching the playoffs on TV. Eventually, only one team will win while the rest of the league can do nothing but watch. That team will be the undisputed winners, and everyone else will have officially lost.
Until draft day. On draft day, every franchise wins. No matter what grades draft analysts give out, the draft gives every franchise hope.
Every team, every fan, has this hope that their franchise will land the next great player. Everyone has the same dream, that their fourth round draft pick will turn into a Hall-of-Famer. Yet they have little idea what that fourth round draft pick is actually worth. For years, teams used something like the chart Jimmy Johnson developed as the coach of the Dallas Cowboys in the 1980s to value picks against each other.
These values are completely arbitrary: there is no statistical evidence to back up the relative values of these draft picks. There is no reason why the 156th pick is 100 times less valuable than the first overall pick. “The Chart” simply dictates how much each pick is worth. These values also have no grounding in the real worth of the players drafted at a given pick. This system is a ridiculous way to value picks because there is no reason behind the values it gives. There must be a better approach.
Taking data from www.pro-football-reference.com, I have created a much better system that more accurately values each pick in the NFL draft, similar to the work done by Chase Stuart. Pro-Football-Reference uses a metric called Career Approximate Value (CAV) that allows one to compare players across seasons and positions. It is not meant as the ultimate NFL statistic. It is useful for comparing large groups of players across time and positions, which is exactly the objective here. Using data from 1980 through 2005, I analyzed each overall draft pick from 1 to 224 (the 32nd pick of the 7th round in today’s draft). I found the mean, median, and standard deviation of the CAV for each pick from those 25 years, creating one set of data that represented the historical value of each pick. I then found the mean, median and standard deviation for this new dataset to determine the expected value of a normal draft pick. I then used that normal pick to standardize my data, finding the percentage value over average, or Career Approximate Value Over Average (CAVOA), for every pick in the draft.
For example, the first overall pick, historically, has had a mean CAV of 66.7. The standard draft pick had a mean CAV of 15.03. . Thus the first overall pick was 443.39% more valuable than the standard pick. Using this method, I found the CAVOA of every pick in the draft, and then regressed it against overall pick number. The regression equation was with an R2 of 0.91599. The R2 means that the variance in overall pick number explains 91.599% of the variance in CAVOA. Using this equation, I found the expected CAVOA for every pick in the draft.
The CAVOA is the comparative value of each pick versus the normal pick and is based off of real, historical, on-field performance. This non-arbitrary statistic is a massive improvement over the old draft chart. To compare my system with the old one, I transformed the old system into a percentage over average as well. The results are below.
The old system massively over values the earliest picks and significantly undervalues mid-to-late round picks. The regression line is clearly a better predictor of future value than the old chart.
Quarterback Josh McCown
But what do these numbers actually mean in practice? The 94th pick is as close to the normal pick, having a draft value of 100.3 and a CAV of 15. If Josh McCown had retired instead of signing as Caleb Hanie’s backup with the Chicago Bears last week, he would have ended his career with a CAV of 15. So if a team uses the 94th overall pick, it should expect to draft a player with a comparable value to Josh McCown: a marginal backup who may spend some time in the United Football League, but will also have some productive outings in the NFL.
In contrast, the first overall pick has a value of 494.6, almost five times greater than the
Safety Rodney Harrison
94thpick; so one should expect a player with a CAV of 74. Brad Johnson, Rodney Harrison,and Corey Dillon all had CAV’s of 74. Given the first overall pick, that is the kind caliber player one should expect to draft: Pro-Bowlers who can help lead their team to a Super Bowl victory. One cannot expect to get a player like Peyton Manning, although it is a possibility; instead, one should anticipate a player of Harrison, Dillon, and Johnson’s caliber.
The distribution around each pick is also interesting. The standard deviation of the expected CAV decreases as the pick number increases, suggesting that picks become less and less variable as the draft continues. While this conclusion is accurate, the coefficient of variation is a better measure of variance than standard deviation in this case. Because there is a separate data set for every pick, one has to compare the standard deviation compared the average of each data set. While the standard deviation decreases as the draft continues, the coefficient of variation is constantly increasing, suggesting that picks become much more variable relative to their mean as the draft continues. This finding suggests that one can be more certain of an earlier draft pick’s value than of a later draft pick.
That said, it is impossible to predict with any accuracy what a given pick’s CAV will be, as the standard deviations are very large. Below is a graph of the 95% confidence interval for every draft pick. The real confidence interval would include negative values for a number of picks, but since this is impossible, I list them as 0. Each pick’s confidence interval includes 0, meaning that no matter where in the draft one picks, there is always the risk of the player picked being a complete bust. However, the upside of picking a superstar diminishes rapidly as the draft unfolds, making earlier picks extremely valuable. This subject, however, is for another study.
This study is the tip of the iceberg of analysis that is now possible to do. There is a ton of work to be done now that we have an accurate representation the value of every draft pick in terms of both on-field value (expected Career Approximate Value) and in terms of other draft picks (Career Approximate Value Over Average). One can use these data to evaluate any trade involving draft picks, come up with a draft strategy that maximizes total expected Career Approximate Value, and much more.
harvardsportsanalysis
David Roher
akcohen
dadler3
jakefisher723
emfryman
ncurran12
arirubin19
John Ezekowitz
wmcmillan
akoenig14
Ben Blatt
Daniel Granoff
Chris Bruce
kevinmeers
Wonderful work!
I did this same analysis for the NBA draft, which you might be interested in, over on the APBRmetrics board: http://www.apbr.org/metrics/viewtopic.php?p=1026&sid=337df09b2f422f57cee41fecbecd9d36#p1026
I should have plotted coefficient of variation, like you. I plotted out a smoothed percentile chart, also, which you might be interested in.
Pingback: The Biggest Over and Under Achievers in the NFL | Harvard College Sports Analysis Collective
Like to see this data broken down by position. My guess is that there is a sweet spot for drafting value at each position. Then you can start to look at this data from a need and value basis when drafting and trading picks.
Great idea. Definitely worth follow-up.
Interesting to look at the famous Rivers-Manning trade. According to the old system, the Giants got 3000 points in exchange for 2219 (#4 and #65 in that draft, and what turned out to be #12 and #144 the year after).
According to your system, they got 494.6 in exchange for 855.4.
I think your system is closer to thr truth. By practically any sensible measure, San Diego got the best of that trade.
But the Giants won a Super Bowl and Rivers doesn’t look like he’ll ever be good enough. Sometimes stats get in the way of reality.
The missing component to valuing picks is cost. The new rookie wage scale drastically reduces the cost of the top picks which increases their relative value compared to drafts conducted under the prior CBA where a top pick could be so expensive that teams may actually have preferred a lower pick. Not taking anything away from the awesome work you’ve done here though. It still provides a way to forecast the output from the picks which is more than we had before.
Very good point–the new rookie wage scale definitely changes the complexion of the draft. However, I don’t think that point really changes much with Kevin’s model. This model is not exactly like the Massey, Thaler paper, which calculated ‘excess value’ (i.e. [free market value] – [rookie wage value])*. Here, Kevin is only measuring Approximate Value, which is basically the same as ‘free market value’ (you can also think of this as production). So when he says that pick 18+19=1, there is no attention paid to the salaries of those relative picks.**
Now under the old rookie wage system, if the ‘free market value’ of 18+19=1, then according to Massey, Thaler (and common sense) a team would rather have the 18+19 picks (since the salaries of those two picks combined are far less than the first pick and you are getting the same production. Under the new system, the number one pick looks a lot better (although probably still not preferred to 18+19) because you do not have to pay nearly as much as before.
So, you are correct that the new rookie wage scale makes the higher picks more valuable than they were under the old scale (the production should be the same, the wages are lower, hence the excess value is higher). However, the higher picks are likely not more (excess) valuable than Kevin’s chart. Those picks are still paid substantially more than picks later in the draft. A good way to test this, would be to convert AV to $’s and then subtract actual contract dollars to really know the excess value of each pick.
System design note: If a draft is trying to accomplish the goal of talent distribution to the worst teams, then the excess value should be constantly decreasing through the draft (so that the worst teams receive the most value). The ‘right’ rookie wage scale would have a similar shape to the way the productivity falls throughout the draft–if wages dropped faster than productivity, that would mean that for some pick X, pick X+1 (later than pick X) would provide more excess value.
*I believe the Massey, Thaler paper only looked at excess value that a player provided on his initial contract, since that is what a team essentially gets from a draft pick (after that, they are paying market rates). Maybe the next iteration for Kevin is to produce a chart that shows approximate value in the first contract (which is tricky because rookie contract lengths vary). To truly value the picks, this would be necessary. Even this approach has flaws since ‘market rates’ is a bit of a misnomer…the best players rarely hit the market (see Manning, Peyton and basically every good quarterback not named Brees).
**One last Massey, Thaler point: they went so far as to claim that the top pick was actually worse than a late first rounder (so 1<28). Based on their calculation of excess value, they were right. Of course, excess value is not the only thing a team should look for in the draft; they also need real production. In recent years, the salary cap has not really restricted teams from spending, so they really all could spend more. If teams were right against the cap, the goal would be excess value…when there is space under the cap (from a sport, not business standpoint), teams should be fine paying 'market rates' for production.
But doesn’t this system ignore the fact of WHO is drafting? Bad teams are the ones drafting early in the round. But they may be bad teams because they make poor personnel choices, (i.e. they draft badly). So the first pick may not be as valuable to the Cleveland Browns (historically bad drafters) as it would be to the Pittsburgh Steelers (historically good drafters). And, therefore, the values being assigned are skewed by who was drafting when, rather than what a pick should be worth.
In other words, Tom Brady increases the perceived value of 6th round picks because he’s a Hall of Famer. But really, the 6th round pick isn’t that valuable, everyone in the league just valued him too low.
I think your data is more valuable as a reflection of how well (or not well) the average team drafts in each round. It doesn’t really reflect the potential value of each pick in the draft. The value of a draft pick is always in the opportunity to get a great player. (In other words, if a team trades for the first pick, they give up a lot because both teams are assuming that the first player taken will be a superstar. While your data shows this may not be the case, the team giving away the 1st pick should still be compensated as if it will be. It’s not their fault, after all, when the other team trades up for #1 and then chooses poorly.)
I really enjoyed this article, though. Interesting stuff!
Very interesting points. I agree that this model is more descriptive than predictive. However, there is still a lot that we can learn from this model.
One counter-hypothesis: maybe the top pick values are artificially deflated because the “poor” players picked at the top of the draft go to crummy teams and do not receive support, while those picked later in the first round play with great supporting casts and do much better. If your theory were really true, wouldn’t we expect to see it hold up in later rounds as well? Under your theory, the approximate value of curve would trend up (or at least the first derivative would be positive) toward the end of each round as the “smart/good teams” made their choices.
A note about the draft: it is important to keep in mind that the draft is by definition a “Winner’s Curse” situation in which the team that drafts a player places a higher value on that player than any other team (otherwise somebody else would have traded for that pick).
This will be interesting to see at play with Andrew Luck. In theory, a team like the Colts should not keep the pick because they still have a pretty good option in Peyton Manning (if he is healthy). Shouldn’t a team like the Jaguars (sorry, Gabbert) trade away their next dozen first round picks to get Luck? Here’s a better way to view it: if the Colts did not have the first pick, would they consider trading for it? The answer is probably no…that should also mean that they should not keep the pick…unless no team is willing to offer anything close to their true valuation of the pick. This is a little bit of the status quo bias (lower drafting teams afraid to make a move for fear that it doesn’t work and the Colts afraid to make a move for fear that Luck is amazing) and the endowment effect (valuing what you have more simply because you have it).
Great article. As mentioned above, I would love to see the analysis in terms of actual money valuation of the players (and how it has changed with the rookie wage scale) because that’s a huge factor in the minds of team management. I would guess there are (or at least used to be) huge inefficiencies in the market.
I’d be interested to see the thirty-two regression curves for all 32 teams since the latest expansion, to see which teams are real draft savants. The spike in average value around the #32 spot could be noise, but it could also imply that a team who can win a Super Bowl is also good at turning rookies into productive NFL players.
What about an analysis that talks about maximizing the amount CAV on field. This comes into play with the Luck-Manning scenario. It’s great that you could have two QB’s with massive CAV’s or expected CAV in Luck’s case, but in the end it doesn’t improve the product you put on the field. (Until Manning retire’s of course.)
Which brings up another idea for analysis. Future Value versus Present Value for players. This is something we see in the draft frequently, teams value winning now, and therefore value players that can help them win now. Not to mention the incentive system for coaches pushing them to win now. Therefore if a “Discount Rate” for players could be found that could actually value them with time as factor I think that would be extremely valuable.
Really great stuff. I’m convinced that certain teams (read: Patriots) are way ahead of the curve on the upshot of this analysis, as evidenced by their consistent habit of trading down on draft day. It will be interesting to see if other teams catch on.
I’d also be curious to see the pre-and-post regressed numbers on your chart, mostly to eyeball the variance by pick for those of us not well versed in R^2, as well as to maybe find a better fit on the tail. In eyeballing it, it appears that there may be even LESS difference between pick 150 and pick 224 than your curve would suggest.
Furthermore, aren’t 7th round draft picks predisposed to getting shafted on personnel and roster decisions early in their career? Once you consider the fact that a GM is a lot more hesitant to cut a 5th round draft pick in training camp than a 7th rounder (thus allowing him to play more and rack up more CAVOA), I’m guessing that the two are practically identical in ability.
Excellent work.
Pingback: NFL Draft Efficiency Before and After the Rookie Wage Scale | Harvard College Sports Analysis Collective
One think that would be interesting would be to see a CVAOA with lets say the top 20,10 and 5 percent of those picks value wise and see how the trendline changes.