Editor's note: To see the detailed stats compiled by ZEUS, go here.
After the Patriots lost Super Bowl XLII by a mere three points, sportswriters everywhere began to look back on one particular play that might have cost the team their perfect season. After an impressive opening drive in the third quarter, the Patriots, then leading 7-3, stalled at the Giants' 31-yard line. Instead of sending in Stephen Gotkowski to kick a field goal, though, Coach Bill Belichick left his offense on the field to try to convert on 4th and 13. Tom Brady's pass fell harmlessly to the ground, and Belichick's reputation as strategic genius took a hit.
"We'd like to think that if you have technology that does something better than someone else does it, it's going to be adopted."—Frank Frigo
Even though the Patriots lost possession and eventually the game, it turns out that Belichick made the right play call. In fact, by calling for a pass attempt, he increased his team's chances of winning ever so slightly. How can we know that? Because of a sophisticated modeling program called ZEUS. Developed by Frank Frigo and Chuck Bowera couple of champion backgammon playersZEUS is a simulator that can take any critical play-calling decision in any NFL game and tell you which option results in the highest chance of a particular team winning the game. At the fork in the road that is the coach's choice of play calls, ZEUS can simulate hundreds of thousands of games based on the various possibilities. As it turns out, in the simulations where Belichick called for a pass as opposed to a field-goal attempt, his team won 1.2% more often.
While the press is busy skewering coaches for what turn out to be good play calls, they continue to overlook the boneheaded calls at critical junctures that add up to deprive teams of, on average, .85 games per year. Coaches continue to punt when they shouldn't, kick field goals when they should go for touchdowns, and kick extra points when they should go for two. Conventional wisdom and conservative play cause even the smartest coaches to hurt their teams' chances of winning.
At the end of last year, Frigo and Bower compiled ZEUS's rankings of the NFL teams for ESPN by aggregating the coaches' play-calling errors over the regular season to see how badly they had hurt their teams' Game Winning Chances (GWC). This year, they calculated the rankings for Gelf, and included a separate calculation of what they call hyper-critical calls when the team was facing fourth down with less than five yards to go. What they found was startling. Every team lost somewhere between 0.42 and 1.42 games due to poor play callingincluding the Giants, who ranked near the back of the pack with an aggregate 1.1 games lostand teams chose the correct fourth-and-short call less than 50% of the time.
Gelf talked to Frigo and Bower about ZEUS's rankings, why coaches continue to make bad decisions, and why ZEUS hasn't gained a foothold in NFL culture. The following is an edited transcript of the phone interview.
Gelf Magazine: Every year, you've calculated, coaches cost their teams somewhere between a half-game and a game and half with poor play calling. Is this a recent phenomenon?
Chuck Bower: We haven't really looked at other years, but we see no indication that they've gotten any smarter or any dumber. I think this is the way they've always been.
GM: Let's take a look at [San Francisco's] Mike Nolan. Last year he cost his team a game and a half, but this year he only cost his team a half-game. Has he learned, or did he simply have fewer chances to mess up this year because his team faced fewer of those decisions?
CB: We customize teams to the quality of their players. To throw the ball on fourth-and-medium or -longthey wouldn't have done very well. ZEUS picked up on that and based its expectations on that.
Frank Frigo: You have certain situations where one play call may be right for a more proficient team that would be incorrect for a poorer team. If you're not Peyton Manning or Tom Brady, you're probably inept at fourth-down-and-five conversions.
GM: The Buccaneers last year were terrible and this year they were actually pretty good. The coaching decisions that they made went from being respectable last year to pretty terrible this year. So if you have a bad team and you're making these bad play calls, you don't look quite as bad to ZEUS?
CB: There's another correlation there. At least in the Game Winning Chance calculation, if a team is either way ahead or way behind in the game, whatever decision they make isn't going to have a very big impact because it isn't likely the team that's behind is going to catch up. In those particular situations, a bad decision could be hidden.
FF: You can have a very clear error that has a very small magnitude. An error gets magnified late in the game with a tie score or in overtime. You tend to get bigger GWC amounts on those types of decisions than you do during lopsided scores, whether you're up or down.
GM: There's not that much correlation between coaching reputation and where the teams rank in the ZEUS calculations. It looks like nobody is paying attention to these critical calls.
CB: I think that's the bottom line. Even your best and brightest guys are making these mistakes.
GM: Let me get this on record. You think Tom Coughlin is an idiot, right?
CB: No comment.
GM: Have you modified the model since last time we talked?
CB: One of the things we definitely did was to improve the modeling for third and fourth down near the goal line. The defense plays differently on those downs and near the goal line. The success rates definitely vary depending on what down you're on and where you are on the field. We modified the model to be more realistic.
"Clearly, some of these coaches must know that they're doing some things wrong because they're conforming to the crowd."Frank FrigoGM: Do you feel like a stats-based approach to coaching is gaining any traction in the NFL since last time we talked? You guys have gotten lots of press, as have other stats programs like AccuScore. Are people in the NFL more open to this?
FF: Doesn't seem like it's reflected in the numbers of errors.
CB: Our phone hasn't been ringing
FF: A few years back, we were actively marketing to the NFL. They were sniffing around a bit. We were getting asked to look at stuff. We were willing to do a bit of free analysis, but once we came to the realization that they weren't going to step up, I guess we sort of got less enthusiastic about marketing.
GM: So you don't think a team's going to exclusively license ZEUS?
FF: We don't know.
GM: Why couldn't you do something like what Football Outsiders is doing and what some of these other sites are doing, which is turning it into something for the fans as opposed to something for the teams?
FF: We've been considering a number of different possibilities. We may give fan access to the website. It's been kind of an intellectual endeavor to see what we can learn. We'd like to think that if you have technology that does something better than someone else does it, it's going to be adopted. History tells us that's the case. People don't move right away. We need an owner who understands us and is wiling to build an organization around it. That's the ideal situation. Someone who's not afraid to adopt.
GM: Are there other places you could prove it workssay, college footballbefore it goes to the big leagues?
FF: It's really designed specifically for the NFL, but it could probably provide a lot of valuable information for the college game. There are a lot of backwards coaches in college football, but there are probably a lot of cutting-edge guys who don't have the pressures of the salary. We've thought a lot about the psychological biases of why coaches aren't adopting some of these tactics. Clearly some of these guys must know that they're doing some things wrong because they're conforming to the crowd.
GM: When we talked last time, you mentioned that Marvin Lewis told you that sometimes being uncontroversial was more important than winning.
FF: Without question. The other thing that you've got is this fraternity of coaches that came up through the ranks and are really judged by the year-end statistics. ZEUS doesn't care how much you lose byit's just trying to optimize your winning chances. If you're a defensive coordinator, you care a lot if you have the worst defense in the NFL because you'll never get a head-coaching job. These guys guard their statistics very closely. There's probably some of that bias in there. Some things that are theoretically correct don't necessarily serve their own interests.
GM: Can you take the guts of ZEUS and make them applicable to other fields?
FF: We've learned a lot from this process in terms of how to model a game and pick out inefficiencies. We're taking these methodologies and looking at other markets and other games.
GM: What else are you guys looking at?
FF: We've thought about poker a little bit. We've thought about commodities and equities markets a little bit. We've thought about sports gaming a bit. There are a lot of different fronts where this stuff could be potentially applicable.
Related in Gelf
Find out exactly where your team ranks in play calling abilities. ZEUS has the details here.