Equally Inaccurate: An Analysis of Mel Kiper Jr. and Todd McShay’s Draft Rankings

By Kevin Meers and Scott Sherman

Conan vs. Leno.  Jack Shephard vs. John Locke.  Seinfeld vs. Newman.  Over the years, hundreds of great rivalries have come to dominate our television screens.  But to sports fans, none of those individual competitions are quite as captivating as the one between ESPN’s two draft gurus: Mel Kiper Jr. and Todd McShay.

Image

ESPN Dallas

There are many iconic hairstyles in the National Football League. Brett Keisel’s lumberjack beard. Troy Polamalu’s bodacious curls. Clay Matthews’ scintillating golden locks. But among the most distinct looks is that of Kiper, whose hair, glasses, and those squinted eyes are one of the signature images of the NFL draft.  Every year, he and his fellow heavyweight, the former University of Richmond waterboy McShay, slug it out on millions of television screens in the months leading up to the draft a dramatic attempt to display their expertise.  But which of these two titans is a better judge of talent?  This post sought to answer that question.

To compare the success rates of the two gurus, we collected the rankings of the Top 25 players in each year’s draft from 2006 (when McShay joined ESPN) through 2010.   Kiper’s rankings were determined by his final “Big Board” before each year’s draft, while McShay’s were determined via Scouts Inc.’s final Top 25 (McShay runs Scouts Inc.).  We then compared each player’s ranking to their actual performance ranking in their draft class (performance here is approximated by Career Approximate Value).  As an example, Kiper ranked Julius Peppers as the best player in the 2002 NFL Draft, and that prediction proved to be dead on, as 10 years later Peppers has the highest CAV of any player from the ’02 draft.  While we would have preferred to use the ranks of all players drafted, Kiper only releases his top 25 prospects on his “Big Board,” which severely limited sample size.   Nonetheless, here’s what we found:

Image

            As the chart above shows, Kiper and McShay have proven to be equally inefficient judges of talent over the past five years.  Neither expert’s rankings were a particularly good predictor of how a player would perform compared to the rest of his class.  In fact, the average errors listed above suggest that the gurus’ rankings are off by around 35 spots when compared to the player’s actual performance.  In other words, a player Kiper or McShay ranked as the 15th-best player in the draft is most likely to actually have been the 40th-best based on CAV to this point in their careers.  The root mean square errors (RMSE) imply that Kiper is very slightly less prone to extreme errors than his ESPN counterpart, but that difference is not significant.  RMSE severely punishes large errors, so having a comparatively lower RMSE implies that one makes fewer huge errors.  However, like the average errors, the RMSE for each scout is enormous.

Indeed, when looking at the lists themselves, it is clear that both Kiper and McShay have, like all draft experts, been prone to extreme hits and misses when compared to the other.  In 2006, both had future busts Matt Leinart, A.J. Hawk, Vince Young, and Michael Huff ranked in their Top 10s.  But McShay was wise enough to include future All-Pro center Nick Mangold in his Top 25, while Kiper instead opted for Jason Allen.  The following year, McShay found more success by ranking Darrelle Revis tenth overall, but the Island himself was nowhere to be found on Kiper’s Big Board.  Also that year, both Kiper and McShay smartly pegged Calvin Johnson, Adrian Peterson, and Joe Thomas as three of the four top players in the draft, but both also had JaMarcus Russell and Brady Quinn in their top sevens.

In 2008, Kiper wisely ranked future bust Derrick Harvey 23rd—McShay had him 11th—but Kiper had fellow pass-rushing disaster Vernon Gholston 7th, compared to 15th for McShay.  In 2009, both had the bust triumvirate of Aaron Curry, Michael Crabtree, and Jason Smith in their top four, but Kiper pegged Matthew Stafford at No. 3 while McShay had him at No. 7.  McShay, however, wisely had both Clay Matthews and Hakeem Nicks in his Top 25, while Kiper had neither (but did, unlike McShay, include Percy Harvin).  Finally, in 2010, Kiper, unlike McShay, smartly deemed Ndamukong Suh a better prospect than Gerald McCoy, but that success was immediately negated by his ranking of Jimmy Clausen at No. 4 overall (McShay, to his credit, did not feel Clausen was a top 25 talent).

Clearly, then, both “experts” have been both on-point and wildly off-base with their draft predictions over the past few years. On the whole, neither’s rankings are very accurate.  If a player is listed in the Top 25 by either scout, that players will, on average, be in the top forty or sixty players from their draft class, but he is absolutely not guaranteed to be one of the very best players from his year.  Furthermore, none of the differences between the average errors of Kiper and McShay are statistically significant: the differences between them are likely more due to luck than actual skill.  In other words, you could take Kiper’s “Big Board” or McShay’s Top 25 and ask any random person to put them in any random order, and on average, that person’s rankings will be about as accurate as predicting future NFL success than any of ESPN’s two experts.

Of course, this study was limited by its sample size, as we’d have a fair better chance of really determining who was more accurate if Kiper and McShay both ranked every player in the draft (which only Scouts Inc. does; Kiper merely publishes his Top 25).  But the nearly-equal numbers in the table above suggest there is not much of a difference between either’s ability to form an accurate Top 25.   Of course, predicting NFL success—as it is in any sport—is extremely difficult, and it’s certainly possible that Kiper and McShay are statistically more accurate than any of the dozens of other websites that publish draft rankings each year.  Since player rankings tend to be generally similar, it’s likely all such sites would be prone to a number of mistakes.  But to determine whether such errors were as significant as those made by Kiper and McShay, future research should compare ESPN’s two gurus success rate to that of their lesser-known counterparts around the web.  Until then, the best we can say is that neither Kiper or McShay is extraordinarily good at what they are paid to do, or even better than the other at doing it.

About these ads
This entry was posted in NFL Football and tagged , . Bookmark the permalink.

27 Responses to Equally Inaccurate: An Analysis of Mel Kiper Jr. and Todd McShay’s Draft Rankings

  1. snl1010 says:

    AJ Hawk may not have lived up to expectations because of where he was drafted, but I don’t think you can say that someone who has played in the league six years and averages 99 tackles a season is a bust and compare him to Matt Leinart and Vince Young.

  2. Great study. Both of these guys are just there for the entertainment value, a role which they succeed at. It’s funny how the media brands a guy an expert and everyone accepts it.

  3. dadler3 says:

    Nice work. I’ve wondered whether ESPN encourages Kiper and McShay to disagree on players purely for entertainment value. It was more interesting television because one guy loved Tannehill and the other thought he was horrible than if they both loved or hated him. This of course is the horrible formula for Stephen A. Smith and Skip Bayless’ show. On the other hand, ESPN may just choose to focus on those places in which Kiper and McShay disagree.

  4. herringbonesports says:

    Very nicely done. It’s somewhat amazing that Kiper doesn’t publish anything outside his Top 25 when hes he’s done the homework and has an opinion on every single pick into the 7th round.

    Great data here though and it certainly validates the thinking that claiming draft winners off what the “experts” fed us isn’t a great precursor to team prospects. Tough seeing my boys Quinn & Clausen on here. I can defend Quinn deserving another chance with NFL stats all day … Clausen not so much despite one of the greatest statistical seasons in Irish history.

  5. rich says:

    They are both buffoons who clearly know nothing about success in the pro game. They have these moronic mock drafts that real personnel people laugh at. When teams draft differently then those morons predict it is invariably called a “reach” or a surprise. They should wear pointy shoes and court jester hats when they give their opinions.

  6. AnotherAndyR says:

    Accuracy is a relative concept, so it would be nice to have a broader context for evaluating your claim that Kiper and McShay are “inaccurate”. How do they compare with other draft gurus? Are NFL teams significantly better at predicting player performance (i.e., how well does performance correlate with the order in which players are drafted)? Are NFL careers harder to predict than other major sports?

    I hope you are not implying that a “random person” could predict player performance just as well as Kiper and McShay. Your example seems to disregard a major aspect of their predictions: the pundits do not merely arrange a predetermined list of 25 players; they choose those players from a much larger list of eligible draftees. Random selections from the larger list would undoubtedly be less accurate.

    The Huddle Report uses different criteria to compare the predictions of many draft pundits, including Top 100 Big Boards by both Kiper and Scouts Inc. It would be great to see a collaboration that combines their extensive database with more sophisticated statistics.

    • Roy Sahr says:

      Thank you. I was going to point out the same things if somebody else hadn’t. The methodology appears completely lacking here to draw any valid conclusions. I’d also want to know what other variables are accounted for when determining their ability to predict a prospect’s future performance. What role do unpredictable events such as injuries play in their analysis? This whole thing just seems way too thin on substantive analysis for something calling itself The Harvard Sports Analysis Collective.

  7. Goldboink says:

    The main problem with all of these draft experts is they can’t possibly do all of the work that is required to actually scout these players. NFL teams have highly skilled and well organized scouting departments. Mel Kiper is basically sitting in his home office collecting stats and press clippings and coming up with a product that can be monitized. Nobody checks his accuracy among his peers because they variously use teams of monkeys with typewriters, an octopus, an ouija board or blog posts to formulate their draft boards.

    The agregating of available public information includes a lot of stories generated by the athletic departments of the universities who are also trying to monitize their product. Consensus picks are mixed in with players who were hyped by thier universities or have some buzz spread by enterprising agents but are weeded out during the combines and interviews and analysis and were either picked low or flamed out. At the end of the day having the kind of error rates may rank them above Nostradamus as soothsayers or predictors of the future. I wonder what Gayle Sayers CAV was.

  8. Pingback: Steelers Draft Needs: Mocking Mock Drafts » Black & Gold Forever

  9. ben1332 says:

    Did you really just group Michael Crabtree with Aaron Curry and Jason Smith?
    That makes me question every word I’ve read.
    As soon as he was allowed to play with a quarterback that wasn’t named Smith, he became one of the top-10 receivers in the league.

  10. crs98 says:

    I liked this analysis, but it’s really missing the obvious control group of the actual order in which the players were selected by the NFL. Are either McShay or Kiper any better (or worse) than the NFL as a whole at predicting performance? It’s interesting to compare the two pundits, but it’s even more interesting to compare them to the NFL as a whole. My best guess is that we would find they are no better or worse than the actual NFL, but it would be really interesting to see this analysis added.

  11. ira haron says:

    does anybody remember Joel Buchsbaum ? google him and read a little. …

  12. Chad says:

    “In fact, the average errors listed above suggest that the gurus’ rankings are off by around 35 spots when compared to the player’s actual performance. In other words, a player Kiper or McShay ranked as the 15th-best player in the draft is most likely to actually have been the 40th-best based on CAV to this point in their careers.”

    This should say “50th-best”. With this basic math being incorrect, it makes a person question the validity of the rest of the math used to create this article.

  13. Ray Pleasant says:

    Both Kiper and Mc Shay seem to have a racial bias against African-American QBs in the draft.
    Case and point is their projection of Cam Newton and Blaine Gabbert. Newton’s 2010 stats were
    very impressive. But, Blaine Gabbert’s stats were very pedestrian, even with the weaker defenses
    in the Big 12. This year the two so-called experts made a very big deal about E.J. Manuel’s draft
    position. I didn’t hear the same criticism with Manuel’s predecessor, Christian Ponder’s whose
    college stats were very pedestrian and did not warrant his high draft selection. His NFL stats
    have been mediocre at best. Gabbert’s NFL stats have been terrible. Both Gabbbert and Ponder may perform better in 2013. But, it remains to be seen.

  14. Steve says:

    There are way too few articles like this. The networks should do exactly what was done here, then put the best draft prognosticators on the air. Sure, if they have no air presence, you can’t use them. Still, there have to be DOZENS better than Kiper or McShay. C’mon ESPN!

  15. Pingback: Big Brother 15: Top 3 Pre-Season Picks

  16. Pingback: Chiefs, QB Trends, Kiper & McShay - Arrowhead Addict - A Kansas City Chiefs Fan Site - News, Blogs, Opinion and more

  17. Joe D says:

    I dpnt understand how people can act like they know how a player’s future will turn out in the nfl. You can determine how athletic a player is by the combine and examine the game tapes to see how well they did at the college level, but you can’t get inside their heads. A guy could have a high motor and great work ethic prior to entering the nfl, yet all that could change once they sign that contract.

  18. Peter D. says:

    There is no reasonable way to classify Michael Crabtree as a “bust.” While he may not be Calvin Johnson, he’s far from a “bust,” having ranked in the top 15-20 WRs in the NFL consistently, posting an 1105 yard season in 2012, and being the NFC postseason receiving yards and touchdowns Leader in the same year. His 2013 season was cut short by an Achilles injury, but he showed up in the playoffs and did well. Also keep in mind that the 49ers are a run-first-run-often team and he had Alex Smith as his QB for his first 3 years in the NFL. (Love me some Alex Smith, but he’s no Kaepernick when it comes to producing receiving yards for his WRs)

    All that to say it’s unfair to call him a “bust.” He just isn’t.

  19. Pingback: Fantasy football outlook for the rookie quarterbacks in 2014 - statsmagazine.com

  20. Pingback: Fantasy football outlook for the rookie quarterbacks in 2014 | Headlinenews.org

  21. Pingback: Fantasy football outlook for the rookie quarterbacks in 2014 - I Vote Sports

  22. Pingback: Fantasy football outlook for the rookie quarterbacks in 2014 - Gubta News Media

  23. Pingback: Fantasy football outlook for the rookie quarterbacks in 2014 | Distinct Athlete

  24. Pingback: Fantasy football outlook for the rookie quarterbacks in 2014 | Click India Life

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s