A Statistical Analysis of the Miracles of Tim Tebow

By Chris Bruce and Andrew Mooney

You can find an update of this post including another two weeks of games here. You can also read this post here on Boston.com.

Over the first 5 weeks of the NFL season, the Kyle Orton-led Denver Broncos struggled to a 1-4 start. Then, yielding to fan pressure, they inserted Tim Tebow at quarterback and went 4-1 over the last 5 games in rather spectacular fashion, producing a flurry of ridiculous media coverage, spurring Denver fans to start wearing Jesus jerseys and creating a whole narrative around Tim Tebow’s Miracles. We thought we’d join the fray and quantify just how much of a miracle Tebow’s success has been.

The Kyle Orton era in 2011 was a rather unfortunate one for Denver. Given their points scored and points allowed, a Pythagorean expectation would predict them to win only 36% of their games, and with some close losses, they unsurprisingly fell to 1-4. After starting Tebow and winning 4 of 5, you might expect that the team’s performance had improved drastically, but that’s actually not the case. Tim Tebow’s Expected Points Added (EPA – AdvancedNFLStats.com’s measure of the points expected to come from the plays that a player executes) over his 5 games is more than 13 points below Kyle Orton’s. The Denver running backs’ and wide receivers’ EPA per game also dropped from the first 5 games to the last 5 games, from 0.5 to -3.8 and from 1.5 to 0.4 respectively. The only thing that has improved is the defense, and it hasn’t been by much – Football Outsiders’ Defense-adjusted Value Over Average (DVOA) for Denver’s defense, a measure of the effectiveness of a defense, has gone from an average of 9.5% to 8.7% (lower numbers, including larger negative numbers, are better in defensive DVOA). Given all of this, you would expect the Broncos’ probability of winning to drop even further – had they exhibited that performance over the first 5 games, their expected winning percentage would have dropped to about 20%. Tebow’s success, then, sounds more and more like a miracle.

And just how spectacular a miracle? Using Jeff Sagarin’s ratings for the last 5 weeks, which take into account strength of opponent and location of the game, the Broncos would have been expected to win 1.9 games over that 5 game stretch. So, it appears that Tim Tebow has (miraculously?) pulled an extra 2.1 wins out of thin air.

How can this be? Essentially, while Tebow and the rest of the Broncos have not been producing better numbers, they’ve been producing their best numbers at the right times. This can be seen in the difference between Expected Points Added and Win Probability Added (WPA) for Orton and Tebow. Win Probability Added is AdvancedNFLStats.com’s measure of how much a player’s plays have contributed to the team’s chance of winning the game. For instance, a 20 yard touchdown pass to take the lead as time expires will be worth much more WPA than a 20 yard touchdown pass to take the lead in the 1st quarter because, in the second scenario, there is still a significant probability that the opponent comes back to win. Tebow has a much worse EPA – meaning that he has objectively produced worse results over all of his plays – but a significantly better (though still negative) WPA – meaning that his plays have contributed more to the chances of his team winning. While Tebow’s plays have produced much fewer expected points than Orton’s, Tebow’s positive plays have come at crucial points in the game, when they have a much larger impact on the outcome: think his 20 yard TD run for the lead with 58 seconds left last week, or his 56 yard TD pass for the lead with 6:44 left the week before. Likewise, Tebow’s negative plays have come at points where their effect was less harmful. Similarly, the Denver running backs have also been exceeding their expected WPA over the past 5 games. Is this just coincidence or clutch play and strong leadership by a guy who “just wins football games”? The jury is still out for Tebow, but history does not bode well for Broncos fans.

The vast majority of players who started their careers with similar characteristics (a relatively high WPA given their EPA) eventually regressed towards the mean and saw their numbers come in line with expectations. A very clear relationship can be seen between a player’s WPA and EPA. The chart below shows the EPA and WPA of all seasons for all quarterbacks that have played in the league this year. Using a regression to quantify this relationship, we can then say what a quarterback’s expected Win Probability Added should be for the Expected Points Added during their play – or in other words, how much they should be expected to affect their team’s wins with their plays on the field, independent of timing during the game or score of the game when the plays occur. Tebow’s WPA so far this season is 0.39 higher than one would expect given the Expected Points Added by his plays. Other quarterbacks who started in a similar situation could not sustain this over multiple seasons. The graph below shows the average difference between actual WPA and expected WPA for all current quarterbacks who started their careers like Tim Tebow – with a significantly higher WPA than expected (we took all QBs who had a difference greater than or equal to Tebow’s in their first year attempting over 100 completions – 10 QBs fit the bill and had more than 1 season of play; interestingly this group included top QBs Aaron Rodgers, Tom Brady, Drew Brees and Eli Manning). Without fail, these quarterbacks saw their play converge towards the expected value over the long term (in case you were wondering, the spike in year 3 is due to exceptional seasons by both Tom Brady and Matt Ryan). 

We can also look at another highly successful, mobile college quarterback with mechanical issues who succeeded early in the league, Vince Young. Young was dubbed a “winner” when he came into the league and got off to a solid 8-6 start as a starter in his first year. But after initially affecting his team’s wins much more than you would expect in his first year, his contribution to wins became much more in line with his play over time, as depicted by the red line in the above graph. After going 9-6 in his second year, he was replaced by Kerry Collins in his third year and came under fire from the media for reported motivation issues. Of course, Young had an outlier year in his 4th year, replacing Kerry Collins mid-season and ripping off a 7-3 record in his 10 games despite fairly average numbers. Accordingly he was dubbed a winner yet again, much like some members of the media are already doing this year. Regardless of the media representation of both players, the likelihood of Tim Tebow sustaining his current winning percentage looks rather dire.

Now before you Broncos fans get too angry, these numbers only argue that Tebow’s wins are more likely to become more in line with his play – if his play improves he’s likely to keep winning at a good rate. Or on the other hand, if you believe that Tebow is truly a clutch player that performs better in high impact situations, then he will sustain a higher winning rate with poorer play. But given that Tom Brady and Aaron Rodgers haven’t been able to do that over the long term, I wouldn’t bet the farm on it.

About these ads
This entry was posted in NFL Football. Bookmark the permalink.

45 Responses to A Statistical Analysis of the Miracles of Tim Tebow

  1. tunesmith says:

    This misses some important context.

    First, WPA and EPA give quarterbacks and receivers partial credit for successful pass plays. They do not for successful zone-read and option running plays. In zone-read and option running plays, the quarterback is making a read and decision after the snap, and the defense is reacting to the quarterback’s decisions. The quarterback should get partial credit for the success of the running plays in these situations.

    Second, in many cases, Denver switches play calling philosophy and formation in the more dramatic parts of the game. They open up the playbook and operate out of a spread formation more often. Tebow has much more success in these cases. It isn’t that Tebow is all of a sudden running the same plays successfully that he was screwing up earlier in the game.

    Some of this is Foxball. People talk about how Tebow breaks some statistical models, but I think the same can be said for Fox. It looks like his “keep it close” mentality can at times even extend to when Denver is in the lead. Denver might be placing a premium on ball control and running down the clock. Denver’s play calling is probably more aggressive when they’re down by seven or less, as opposed to when they’re ahead.

    • Chris B says:

      It’s true that WPA doesn’t take into account read handoff and option plays, but since the WPA of the Denver RBs is negative and also above expectation you would not expect this to help Tebow’s stats (by him getting “partial credit”) or for this win performance to persist in the long term.

      And while Denver may switch play calling to more Tebow-friendly plays in the crucial parts of the game, this begs the question why don’t they always do that? Maybe they think that the spread formation plays can’t work all game long so they save them for crucial points, but if that is the case I think it also brings into question how sustainable that strategy can be in the long term. There’s no question that the strategy has contributed to the difference between his expected and actual WPA, but I think the more interesting question for a lot of people is if it is something that can persist.

      • DCJ says:

        Actually, Fox answered your question about why they don’t call more “Tebow friendly plays” earlier in the game. The problem, as he put it, is that there is not a lot of film on opponents defending the spread formation. Thus the Broncos spend much time experimenting with plays to scope the opponents’ reactions, then make adjustments, leading to the slow starts.

  2. franklapidus316 says:

    Does that EPA/WPA relationship really measure “clutch”? What about Rodgers this year, his EPA blows away his WPA simply because they’re permanently ahead, and often by enough that he can’t add to his WPA. If feels like it picks up opportunity more than anything else. If the Broncos have the courtesy of keeping Tebow in for 5 years so we can get a good sample size, I bet he bucks the trend on the graph, if only because he’s probably the anti-Rodgers (Rodgers must be the anti-christ then….nevermind). They’ll probably get stuck with a permanent strategy of just doing everything they can to keep it close until late, and then save some plays they like better for the 4th quarters. So he would simply get more chances over time than the typical QB.

    • Chris Bruce says:

      Well, there are 2 things. The first is that Rodgers has performed below his expected WPA (according to my simple regression), but just barely. You’d expect his WPA to be just 0.09 higher, as opposed to Tebow’s outperformance of 0.39 in just 5 games. The second is that while Tebow may have had more opportunities to make big WPA plays at the end of games, that does not necessarily mean you would expect a higher WPA from him. Rodgers’ WPA per game is still much larger than Tebow’s, he just happens to get it earlier in the game – taking his games from a WP of 0 to near 1 early, then not even participating in most plays once they’re ahead because they’re running the ball. Tebow on the other hand typically struggles through the first quarters of a game, and has (for at least the time being) been taking his games from a negative or near-zero WP to 1.

  3. Pingback: The Death of Logic | The Sports Geeks

  4. Please tell me no taxpayer funds were used for this… ;-)

  5. Joel Fago says:

    The analysis shows the power of prayer!

  6. Albert Einstein says:

    Huh?

  7. Iget Stupified says:

    Cool! I am in my third week of college Statistics and this article made for some excellent reading. Thanks!

  8. Samuel Di Muzio says:

    Being of simple mind, simple observation and enjoying Bronco success:

    Total games with Tebow starting: 6. Total games won: 5 = .833333 winning percentage

    Total games with Tebow starting: 6. Total games lost: 1 = .1666667 losing percentage.

    Based on percentage of wins vs. losses I am expecting the Broncos to win over the Vikings.
    How they win is determined by Fox/Coaches and the team ability to listen well to them and
    to the Vikings. I will reevaluate my simple formula when the opponent has a superior percentage.

  9. Joseph Croasdaile says:

    No Mention of Turnovers
    No Mention of points coming off Turnovers
    No Mention of Time of Possession

    What is a study worth if the data is incomplete? I would say this is an interesting premise, but without factoring more information that had a direct impact on the game I would say it is flawed.

    • Fredz says:

      Of course turnovers are taken into account, how can you think the WPA and EPA wouldn’t take a hit if you turn the ball over, even more if you give away points.
      T.o.P is eventually taken into account indirectly since the moment the play happens impacts the result. And T.o.P is not that relevant anyway, you can have the ball 90% of the time but if you give up a pick 6 every 10 minutes and don’t score, you won’t get anywhere near a W … The most relevant data is taken into account.

      • Joseph Croasdaile says:

        Time of possession is not relevant? TOP is relevant to the defense and the score of the game… I can’t believe you would say it isn’t relevant.

        also you didn’t address points off turnovers…

        Orton (4.5 games)
        8 TDs
        11 Turnovers
        7 x 8 = 56 points
        Points scored from his 11 turnovers = 27

        Since the 27 points were scored due to Orton’s turnovers one could say the following… Orton’s net scoring was 29 points in 4.5 games which is an average of 6.4 points a game

        Tebow (6.5 games)
        10 TDs
        2 Turnovers
        7 x 10 = 70
        Points scored from his 2 turnovers = 7

        Tebow’s net scoring is 63 points which is an average of 9.7 a game.

    • John P says:

      The argument fails to account for several variables and therefore holds no credibility.

      • mountainman says:

        +1 for John P.
        d.f. fail. You can’t cherry pick stats that support your contention without acknowledging those that you discarded and adjust for the appropriate degrees of freedom.
        Also, this is like reading a statistical analysis of dating/mating.

    • Jeff C says:

      You are correct… This was one of the better responses I’ve read so far.

  10. Shelton Corbett says:

    Keep It Simple Stuupid!

    The magic of Tim Tebow can be explained by simple math. Tim Tebow is the only quarterback in the NFL that requires the defense to use two extra assignments in the defensive front. Thus, the read spread option with Tim Tebow as quarterback requires the defense to have 13 assignments. If Tim Tebow takes the snap and reads the defense, thier will always be two missed assignments. That’s the magic behind Tim Tebow! You don’t need a degree from Harvard to figure out that 13 is more than 11. 13>11=Tim Tebow (The magic behind the numbers.)

    • Jordan Rapp says:

      Actually, he’s one of two – the other being Cam Newton. ESPN actually covered this recently in an article by Trent Dilfer, who was (IMO) an under-appreciated QB in his time and his understanding of the game shows in his well written article. Basically, Dilfer points out, with Newton and Tebow defenses can no longer read the lineman to defend against the play action. Fans follow the ball, but D’s read the lineman. But with Tebow and Newton, they have to defend – in a play action scenario – against: RB run, Pass, QB run, which inevitably leads to missed assignments. And both Newton and Tebow are too big to cover with a single small defender. As Shelton said, all of a sudden, you have too many assignments.

      I would say that at least some of the “regression to the mean” cited in the article comes from defensive adaptation to particular players as the result of film, etc. It’ll be interesting to see how defenses adapt to cover quarterbacks like Tebow and Newton and what other new opportunities arise as a result of those changes.

      Here’s the Dilfer article – http://insider.espn.go.com/nfl/story?id=7336193&_slug_=tim-tebow-cam-newton-wave-future-nfl&action=login&appRedirect=http%3a%2f%2finsider.espn.go.com%2fnfl%2fstory%3fid%3d7336193%26_slug_%3dtim-tebow-cam-newton-wave-future-nfl

  11. Jeff C says:

    You folks are for sure very smart… For me though it makes no difference what the stats are outside of the win loss colums and in the age of win now I’d say Tebow is winning and that my friends is all that matters. Peace…

  12. Pingback: Estudantes de Harvard analisam Tebow « BLOG TOUCHDOWN

  13. Bill B says:

    It is commonly appreciated that football is a team sport and it takes a unified, offensive and defensive, effort to win games; and secondly, that one of the primary qualities of excellent leaders, top performers, is creating an atmosphere where other members of the team work harder and better together, and gain overall higher levels of success. It would be interesting to analyze the stats of other key members of Denver’s offense and defense to see if many more players are playing to a higher level – and it does not take much if each member of the team contributes just a little bit more, to help explain the success they are having in the NFL’s most important stat, wins and losses. Water does not boil at 211 degrees, but does at 212 degrees, an insignificant 1 degree. Yet, that 1 degree makes dramatic qualitative change.

    An argument can be made, going back to high school and college, that Tebow is the kind of Quarterback/leader who by his own exceptional effort and desire to constantly improve in every way his performance, and win, makes others around him better because he creates an exceptional example of hard work and effort to get better every day, along with an extraordinary will to succeed no matter how few minutes or seconds remain on the clock. And secondly, because he always gives credit to others on offense and defense for their successes, while being the first to accept responsibility for lack of success.

    There is no reason to believe that Tebow will not continue to have the same success in the NFL, which he has worked hard to attain at every level so far throughout his football career, if he continues to improve his game in every way, which he has demonstrated in the past, including passing (he threw for 88 TD’s in 3 yrs as a starter with a 66+ percent completion rate, among to tops in college ball), and continues to exude that most important of all characteristics of leadership, making others around himself perform at a higher level, combined with an iron-clad will to win, to succeed, especially when the game is on the line.

    • Jay N. says:

      From the article:

      “The Denver running backs’ and wide receivers’ EPA per game also dropped from the first 5 games to the last 5 games, from 0.5 to -3.8 and from 1.5 to 0.4 respectively.”

      As you can see, they did analyze the play of the running backs and wide receivers, and their performance has in fact worsened under Tebow’s command.

      Again, from the article:

      “The only thing that has improved is the defense, and it hasn’t been by much – Football Outsiders’ Defense-adjusted Value Over Average (DVOA) for Denver’s defense, a measure of the effectiveness of a defense, has gone from an average of 9.5% to 8.7% (lower numbers, including larger negative numbers, are better in defensive DVOA).”

      Tebow could have been the reason for the improved play of the defense. Who knows. But, this is a very insignificant margin that the defense has improved by. (For reference, in week 12 the top performers in DVOA for defense was the Ravens at -19.4%.)

      In essence, each team member isn’t really producing more.

      • mountainman says:

        Jay N.
        What kind of analysis have you done on the validity of the metrics you have sited? Do you really think that EPA per game and DVOA are meaningful in any way?

        Do you think that is air that you are breathing? (hint: you should have taken the blue pill).

  14. Susan says:

    Oh you are so right. Why doesn’t anyone else notice that he can throw the ball?!!! Thanks for posting something reasonable and right about our guy.

    I am a Gator who lived in Denver for 17 years until this year. We were there for the Elway years and don’t get why he is so against Tebow.

    • Jolo says:

      I don’t think he is against Tebow, I think he is unsettled on whether Tebow is the long term qb of the Broncos. His comments were “he has to complete more passes” and “the team needs to be better on 3rd down conversions”.

  15. Gator Joe says:

    This is cute. I enjoy implementing science and statistics to make clever associations. However, you’re messing with my boy, so I have some questions:

    In your 4th paragraph; “Tebow has a much worse EPA –… …– but a significantly better (though still negative) WPA …” What were the numbers? Is the difference statistically significant? If so, what is the p-value? Did you assume normality for this distribution and, consequently, use normality tests? If so, why?

    Your regression comparison of WPA vs. EPA makes an obvious statement. Though I like the correlation, the chart explains that as expected points increase, the probability of winning increases. It’s almost like saying that if one team scores more points than another team, then more times than not, they will win.

    Out of curiosity, can you identify the apparently high leverage point in the regression chart (WPA~7, EPA~300)?

    Also, note that nearly 18% of the variance is unexplained in this association. That missing 18% can be explained by Tebow’s awesomeness (or some other random and unforeseen fact that we are missing, but, as I mentioned before, Tebow is my boy and I am biased, so I’m going with awesomeness).

    By the way, as I was writing this, he just won again. Time to update the numbers.

    • Chris Bruce says:

      Joe – the difference was statistically significant to the 90% level, though not to the 95% level (p-value of 0.07, two sided) and the distribution of residuals was normal (hence the use of a t-test).

      Also, the “leverage point” in the graph was 2007 Tom Brady.

      Thanks for the comments and check the new post for updated numbers.

  16. Pingback: Tim Tebow Is Making Me Question My Atheism [NFL Roundtable] | Sponsor szuka

  17. Frank says:

    “10 QBs fit the bill and had more than 1 season of play; interestingly this group included top QBs Aaron Rodgers, Tom Brady, Drew Brees and Eli Manning”

    Interesting set. It seems many are assuming Tebow will converge toward the expected value because his WPA will drop to his EPA (as a way to dismiss this “Tebow Magic”), but it seems that just as often these “Miracle” QB have been able to raise their EPA to the level of their WPA.

  18. I would be interested in knowing how many of you completing your analysis actually played football at a competitive level through high school (i.e. did not ride the bench at a 1A school in North Dakota). Turnover margin in sports is HUGE! Opponent points scored off turnovers is just as critical. John P stating there is no credibility to the argument is just inaccurate, whether you went to Harvard or Heartland Tech. Protecting the ball and time of possession can cure a lot of other ills (such as a throwing motion that looks as if the QB has a broken arm :-) I have been a Bronco fan since birth, past season ticket holder, and not a huge Tebow fan. I will close with a few things you can’t measure with your statistical analysis; heart, guts, the WILL to win, and the ability to WILL those surrounding you to win. A portion of their success is tangibly explained by the numbers, the remaining portion are the intangibles.

    I would love to see you go back and look at the statistical analysis for the 2000-2001 Superbowl Champ Ravens QB’d by Trent Dilfer. No doubt they won the Championship with a stellar defense led by Ray Lewis. However, if my memory serves me right, (not important enough for me to do the research, i will leave that up to you smart guys) that defense was never under pressure due to the offensive turning over the ball. Trent Dilfer (not the most skilled QB around, not mobile at all) managed the game and did not make mistakes turning over the ball. His stats were mediocre much like Tebow’s, but he did not throw interceptions, nor did he make big plays. He simply managed the game well, did not make the big mistake, and did enough to win. Sound familiar?

  19. D. Franklin says:

    Is there any reason why we are not using football statistics to teach children critical thinking and math? Seems like a great way to get the math haters to be math lovers.

  20. Pingback: Tebow is Still Producing Miracles: An Update | Harvard College Sports Analysis Collective

  21. Pingback: On Miracles, Evangelism and Tim Tebow | Ignite

  22. Pingback: More on the Intangible Train, Tim Tebow | SNY Why Guys

  23. Pingback: I Like Tim Tebow. So Does Everyone Else. Can We Move on Yet? « Better Than Voodoo

  24. ryan says:

    This was so awesome, great topic and thread. What happens when you analyze Terry Bradshaw with this?

    I think part of winning and playing QB, and leadership is very psychological (some say even spiritual). Tebow can win (I don’t know at what rate) if he is somehow able to get the defense to play better. Maybe that is part of being the QB and a team leader? And you note about the ‘timing of performance” it completely relevant and not easy to measure.

    It is kind of like home court/field advantage in many sports – in basketball all the courts are the same size. Football the fields are all 100 yards. An aspect of winning comes down to feeling like you can win or being comfortable. There is something completely psychological or at least non-statistical in “home court” advantage.

    I have no statistical background and can barely remember how to do fractions… This was just a great subject. Thanks for writing.

  25. Pingback: Miracles Or Luck? | Tebow Press

  26. Pingback: What Is Tim Tebow’s Future In The NFL | Tebow Press

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s