2017 College Football Retrospective

It is once again that time of year.  Your neighbors finally used up all of their discount fireworks, the mercury has risen into the 90s, the talking heads on the TV are desperately trying to convince you that NBA free agency is important, and (most importantly) the preseason college football magazines have finally started to hit the shelves.  In continuing what is now my annual tradition, I have inputted all of this preseason football schedule and ranking data into a spreadsheet, and I have run several simulations on the upcoming season.  I would like to share with you want I have found.

I can (and have in the past) spend several paragraphs explaining my methodology.  If you are curious about the details, more information is found here.  But, the basic idea is that over the years I have developed power rankings and algorithms that allow me to predict point spreads and victory probabilities for each game as well as simulate the final standings in each conference based on a reasonable number of upsets.  While my methods are not perfect (I rely heavily on the idea that home underdogs are the biggest source of upsets) I do believe that they provide insight in the coming season, especially as it relates to the impact of schedule.  If nothing else, my predictions usually provide something a little different than just saying the top 4 teams in the preseason rankings will make the playoffs. 

Soon, I will post my predictions for the 2018 season.  But, before I do, I wanted to take a look back at my picks from 2017 to see what went right and what went wrong. My full analysis can be found here.  A few years ago, I came up with a new way to calculate the probability that each team in the FBS would win any number of games in the regular season (from 0 to all 12).  But, my original calculation was flawed in that it did not take into consideration that probability that the preseason rankings were wrong. The data show that, on average, the preseason rankings are off by about 15-20 slots per team, which I showed in this entry.  Last year, I tweaked the method to take this uncertainty into consideration, based on historical data.  The result was a "win probability matrix" for each team.  Below, I show the matrix I calculated last year for the Big Ten.

So, how did I do last year?  As for MSU, last year I wrote that, "MSU is most likely to win 5 or 6 games and MSU’s “expected value” of wins (mu in the table) is only 5.45... But, if you want to be more optimistic, the data suggest that there is actually a 9% chance that MSU finishes with 9 or more wins."

So, clearly (just like everyone else) my MSU pick was way off.  But, this did stimulate two questions.  The first question is: Why? Well, obviously MSU was way better than expected, but just being better is not always enough. The other two factors are the effect of schedule and something you can either refer to as luck or execution.  As for the schedule, sometimes the teams on your schedule are either way better or way worse than expected.  I was able to quantify this difference simply by rerunning my simulation using the post-season rankings as opposed to the pre-season rankings.  As for luck/execution, this is basically the idea of how a team does relative to the likelihood of winning each game.  For example, if a team has 4 games on its schedule that are coin flips, the most likely result is they will win 2 of those games (50% x 4).  If that team were to win 3 or all 4, that it either due to luck or good execution.  If that team only wins 0 or 1, it is either poor luck or poor execution.  I was able to separate all of these variables and create waterfall charts for each team.  For MSU, that chart looks like this:

In the preseason, I projected MSU to win slightly less than 6 games.  Based on my analysis, MSU's schedule was actually slightly harder than expected (by 0.3 games, hence the small, negative red bar).  As for ability, MSU preseason rank was 52, but my final power ranking for MSU was 17.  As for wins and loses, I calculated that this accounted for an additional 2.3 wins.  Finally, in order to get up to MSU's 9 regular season wins, it did require a little bit of luck/execution to the tune of an additional 1.4 games. This is a bit high, honestly, as the average is only 0.8 games for "lucky" teams, and MSU had the 8th highest total of all teams.  For comparison, the average impact of schedule (either positive or negative) was about 0.6 to 0.7.

The second question I wanted to ask was "was my preseason calculation that MSU had a 9% chance to win 9 games or more accurate?"  One way to think about this is that MSU should have been in the Top 10% of "over-achieving" teams in 2017.  Well, it just so happens that MSU was 7th (of 130) based on my math, so that checks out.  But, considering I also calculated both a mean and standard deviation in the win probability matrix, it is possible to calculate the number of standard deviations each team's actual win total was from my calculated mean and see what that distribution looks like.  As as example, I said MSU would win 5.45 games plus or minus 2.2.  I reality, MSU won 9 games, which is 1.6 standard deviations from the mean. This translates to a 95th percentile result, which basically checks out. If I compare the final win totals for each team to the preseason mean and standard deviations, I get the following histogram:

While this is not a perfect bell curve, the shape is pretty close. Furthermore, 62% of the teams were within 1 standard deviation of their expected win total, and 94% are within 2 standard deviations.  The tradition bell curve would predict 68% and 95%.  So, basically, it looks like my math checks out.  Going forward, I would expect this year's prediction to also be mathematically pretty reliable.  

So, what did I get right and what did I get wrong in last year's predictions?  Let's start close to home

Big Ten

Last year I wrote: "my simulation predicts that an 11-1 OSU squad would handle the 12-0 Badgers in Indy to claim the B1G title.... the Badgers have over a 9% chance to go 12-0, which is the 5th highest of all 130 D1 teams, and the highest among Power 5 teams."  Basically, I more-or-less much nailed the top of the Big Ten in conference play.  I predicted that OSU would lose at Michigan and not at Iowa, and I also failed to predict the Buckeye's loss at home to Oklahoma, which certainly cost OSU the play-off spot and hurt the conference's reputation in general.

Below, I show the waterfall plots for 12 of the 14 Big Ten teams:

Big Ten East

Big Ten West

Notably, MSU wasn't even the most under-rated Big Ten team last year.  That banner goes to Purdue, who won 4.5 games more than expected, and actually had some bad luck which kept them from 7 wins.  Meanwhile, the most over-rated teams, ability-wise, were Minnesota (-1.1 games), Nebraska (-1.4), and Michigan (-1.5).  As for luck/execution, MSU was tops in this category, at +1.4 games, with Rutgers and Maryland at +1.3.  As for the not-so-lucky, that would be Indiana (-1.6) and Purdue (-1.1).  Finally, as for the schedule factor, as you might guess, this is pretty similar for all teams in the same conference, and most B1G teams were right around -0.4 to -0.7 (suggesting the Big Ten was under-rated as a conference).  Illinois (+0.6) and Purdue (+0.2) were the only schools that had an easier than expected schedule, while Iowa (-1.1) and Northwestern (-1.1) both had some of the tougher than expected schedules in the country (2nd and 3rd tougher than expected in the Power 5).  

It certainly will be interesting to see how these metric change from year-to-year.  For example, based on these charts, teams like Iowa, Purdue, Indiana, and Penn State may have been a little better than we thought, but lost maybe 1-2 more games than they should have based on bad luck and/or schedules.  Meanwhile, MSU may have gotten a little lucky to get that 9th win as opposed to just having 8.  One thing is for sure, though, and that is that Michigan was, in fact, just an 8-win team last year, and if anything, were slightly lucky.

SEC

How about the SEC? Last year I wrote: "Alabama is in good shape to repeat overall and Florida looks to run away with the East by multiple games."  So... my picks here were not quite as good.  But the explanation is pretty clear.  Ironically, I actually pretty much nailed Alabama's season.  I picked them to go 11-1 with a loss at Auburn and to eventually win the National Title, which is exactly what happened.  Where my model failed in the West was actually with Auburn, who I predicted to lose at LSU (which did happen) and also at Texas A&M (which did not, mostly because A&M was worse than expected).  Thus, Auburn was able to sneak into the SEC Title game.  

On the East side, it simply came down to the fact that the preseason rankings were way, way off.  I had both Florida and Tennessee finishing ahead of UGA, but both teams were simply bad last year, while Georgia was much better than expected.  The waterfall charts of notable SEC team are shown here:

One thing is for sure, and that is that the SEC as a conference was over-rated in the preseason.  Only three teams performed above their projected ability: Georgia (+2.0), Mississippi State (+1.0), and Bama (+0.8). While Tennessee (-5.1), Florida (-4.3), Arkansas (-3.5) were the three most underachieving teams in the Power 5.  As one would expect, this impacted the schedule portion of the waterfall chart and over half of the conference (such as A&M at +1.5) had a full one-game bump in W-L, while South Carolina (+2.0) and Vanderbilt (+2.3) had over a 2-game bump.  As for luck, Florida was the least lucky at -0.5 and Bama was at -0.3.  On the other side of the coin, South Carolina, Auburn, and Kentucky all came in at +0.8.

ACC

Last summer I wrote, "The Clemson Tiger's sneaky-tricky schedule will basically cause them to fall short of Florida State and their much softer conference slate while Miami wins the battle of attrition the Coastal Division."  I also picked FSU to lose two non-conference game to Bama and Florida which would ultimately shut the ACC out of the playoffs.  So, basically I got the Miami part and the FSU losing a lot part correct.  I projected Clemson would have trouble with road games at Louisville, VA Tech, and NC State, and just to spite me, they won all three of those games and dropped by far their easiest conference road game at Syracuse.  Meanwhile, Florida State had some early injuries and limped to 6-6, despite being ranked in the Top 4 in all major publications in the preseason.  As for Miami, I had them originally losing to FSU to result in a 4-way tie for the Division, but since the Seminoles were so bad, the Hurricanes beat them and that put them one game ahead of my expectations.  In addition, some of the other projected contenders (GA Tech and Pitt) were a little weaker than expected and suffered couple upset loses that knocked them out of contention.  The ACC waterfall charts for notable teams are shown here:

The waterfall charts also hint that ability-wise, Virginia Tech may have actually been better than Miami last year, but Miami had the edge in "luck."  We will see how that might manifest itself in 2018.  As for other metrics, the ACC in general seemed to be a bit under-rated, as every team except Duke (0.6) had a negative schedule impact.  But, on a team-by-team basis, the actual ability of each team varied wildly from the preseason predictions with BC (+3.7, 3rd best in the Power 5), Wake Forest (+3.7, 4th best in the Power 5, after rounding), and Duke (+1.5) have having much higher ability than expected and Florida State (-3.4, 4th worst in the Power 5), UNC (-2.4), Pitt (-1.2), and Louisville (-1.0) all being at least a game worse than expected.  As for luck, Virginia (+2.3) was one of the luckiest teams in all of college football (2nd overall and 1st among Power 5 teams), and interestingly both Miami (+1.7) and FSU (+1.0) were high on this list as well.  Pitt (-0.8), GA Tech (-0.9), and VA Tech (-1.0) were not so lucky.

Big 12

Last year I predicted the Big 12 would be a battle between the two Oklahoma schools with the Cowboys coming out on top due to their more favorable schedule, the most important of which was the OK - OK State game in Stillwater.  But, in reality, OK State managed to lose three winnable home games, including the rivalry with the Sooners to finish in a 2nd place tie with better-than-expected TCU.  Meanwhile, Oklahoma did stumble to the much-better-than-expected Iowa State Cyclones, but they won (by a total of 12 points) the other two games that I thought might give them problems: the games at K-State and vs. Texas, in part because both teams were not quite as good as predicted.  The notable waterfall plots for the Big 12 are shown below:

As you can see from the schedule bars, the Big 12 was properly evaluated in the preseason, on whole and pretty much all 10 teams had only a small correction due to their schedule.  Baylor's mark of -0.6 was by far the largest in magnitude.  As mentioned above, the main surprise team was Iowa State who won a full 4 games more than expected (2nd best in the Power 5) simply due to "ability."  TCU (+1.1) and Texas Tech (+1.1) were also noticeable better than advertised.  On the other side of the coin, Baylor (-3.0, 7th worst in the Power 5), Kansas State (-2.1), and Kansas (-0.9) were worse than expected.  As for luck/execution, Oklahoma (+1.1) and WVU (+1.0) were at the top of the heap while Baylor (-2.7, worst in the Power 5), Texas (-1.5), and Iowa State (-1.4) were less fortunate.  

PAC 12

Last year I wrote, "Washington is the favorite in the North and Top 10 quality, [but] my simulation suggests USC will come away with the crown," and in general I got pretty close.  In the South, I picked the Trojans to run the table in Pac 12 play but to lose in South Bend to the Irish.  But, I did mention that the road game at Wazzou would be the biggest challenge.  As the season played out, USC actually did lose both those games to finish 10-2 with the PAC 12 South crown.  In the North, I correctly picked Stanford to beat Washington in the regular season but to lose at USC and at Washington State.  But, Washington also stumbled unexpectedly at Arizona State, which ultimately cost them the division title.  The waterfall plots for notable Pac12 teams are shown below:

As the graphs imply, all PAC 12 teams had a positive schedule correction which implies that the conference was over-rated at the preseason as a whole.  A big factor in this effect is that some of the best teams in the conference: USC (-2.3), Stanford (-1.1), and Washington State (-0.9) had negative ability scores, but offsetting positive luck scores: USC (+2.0), Stanford (+0.6), WSU (+1.2).  In contrast, Washington appears to have been as good as advertised last year (+0.1 in ability), but not so lucky (-1.0).  The only teams that were significantly better than expected were Cal (+1.6) and Arizona (+1.3), while multiple teams were significantly worse: Oregon State (-3.5), Oregon (-3.1), UCLA (-2.8), and Colorado (-2.4).  Finally, in the luck category Utah (-1.4) and Cal (-1.3) were the biggest losers while (in addition to the teams listed above), while UCLA (+1.1) was more fortunate.

Notre Dame

In the preseason, I picked Notre Dame to go 10-2 with loses at Miami at Stanford.  The only difference is that Georgia managed to squeek by the Irish in South Bend as well, so the Irish finished at 9-3 instead and outside of the NY6.  Their waterfall chart is pretty straightforward:  It shows that they were a little better than their preseason rank implied, but their schedule was also a little harder than expected, so it was basically a wash.

Group of Five

Last summer, in discussing which Group of Five team would grab the NY6 slot, I wrote, "Most of the preseason magazines say the Bulls of South Florida, and they are probably correct."  While I and the rest of the country had the conference correct, the fact that no one saw coming (as their highest preseason ranking was 54 by Lindy's) is that UCF was way, way better than expected.  As a result, they ripped through there schedule to finish 13-0, including a surprise win over Auburn in the Peach Bowl.  As for the rest of the AAC (such as the preseason contenders USF and Memphis) as the waterfall chart below shows, they weren't really any worse than expected, it is just that entire conference was under-rated.  We will see if that continues into 2018.  I will also comment that my simulation also correctly predicted that USF would "upset" Illinois and Memphis would upset UCLA.

My dark horse team to sneak into the NY6 was Western Kentucky from Conference USA.  Unfortunately, they actually sucked real, real bad last year and only finished at 4-4 in their own, weak league.  The over-achievers here were North Texas and Florida Atlantic who each won their divisions.

The Mountain West is usually a good source for a NY6 candidate, mostly because Boise State is still keeping that league relevant.  Last year I picked Boise win their conference, but lose 3 games.  That part I got right, but instead of dropping a game to BYU, Boise lost to Virginia, and instead of losing to San Diego State in conference play, they dropped a game to the most surprising team in all of college football last year: Fresno State, who won 9 games despite my prediction of only winning ~3.  My preseason simulation only gave Fresno a 0.1% chance of that happening, which suggests that might be a once in a decade occurrence.

Finally, in the MAC, I projected Toledo to win the MAC with a 10-2 record (check), but I thought that they would match up against Ohio and not Akron in the MAC Title game.  The waterfall chart below for the MAC suggests luck had something to due with this, as Akron managed to upset Ohio in the regular season, and in general, was one of the "luckier" teams in the country last year.

(Sorry Sunbelt, I did not go into much trouble looking into your league either last summer or now)

NY6 / Playoff Picks

In my preseason analysis last year, I projected the following NY6 Match-ups:

Sugar Bowl: #1 Alabama vs. #4 Wisconsin

Rose Bowl: #2 Ohio State vs. #3 USC

Orange Bowl: Florida State vs. Notre Dame

Cotton Bowl: Oklahoma State vs. Michigan

Fiesta Bowl: USF vs. Washington

Peach Bowl: Penn State vs. Florida

with Alabama beating Ohio State in the National Title game.

In reality, the match-ups were:

Sugar Bowl: #1 Clemson vs. #4 Alabama

Rose Bowl: #2 Oklahoma vs. #3 Georgia

Orange Bowl: Miami vs. Wisconsin

Cotton Bowl: Ohio State vs. USC

Fiesta Bowl: Penn State vs. Washington

Peach Bowl: Auburn vs. UCF

So, I basically got 6/12 total, which is not too bad.  As you can see, one or two games here or there can have a pronounced impact on this bowl line-up.  For example, if Ohio State were to have beaten Oklahoma back in September or if Auburn would have stubbed their toes in one other game (thus sending Alabama to the SEC Title game to face Georgia early), my scenario would have been much closer to the reality.  It also would have been a bit closer if teams like Florida, Florida State, and Michigan would not have been so over-rated (not that I am complaining).  

But, that is the fun of college football: it is unpredictable.  But, that does not stop us from trying.  For my next post, I will try to do just that: use my math to predict the 2018 season.  Stay tuned.