2018 Prediction Retrospective

Now that I have released my projections for 2019, I thought that it might make sense to take a quick look back my projections for the 2018 to see what I got right, and what I got wrong.  I had a few questions about the accuracy of my methodology.  So, let's take a look, starting close to home.

Big Ten

2018 projection:  Ohio State had the best raw odds (38%) in the East, but my simulation saw a 2-way tie between MSU and Penn State, with the Lions winning the tie-breaker.  In the West, Wisconsin had the best raw odds (66%), while my projection really liked Iowa to win the West

What actually happened:  Ohio State and Northwestern (preseason odds = 11%) won their divisions

What went wrong:  Penn State failed to beat any other good team in the East, despite a friendly schedule and MSU was a lot worse than expected.  Out West, Wisconsin was worse than expected, Iowa lost a lot of close games, while Northwestern won all their close games.

SEC

2018 projection:  Alabama had the best odds (56%) in the West and the simulation predicted them to win the West with just a loss at LSU.  Georgia had the best odds (also 56%) in the East, but my simulation liked South Carolina as the surprise winner.

What actually happened:  Alabama ran the table in the East, while Georgia won all their games except the road trip to LSU (which I projected correctly).

What went wrong:  South Carolina failed to upset Georgia, and Alabama trucked LSU, but otherwise I did OK.

ACC

2018 projection:  Clemson had the best odds (69%) in the Atlantic Division, and my simulation picked them to win the Division with a lone loss at Florida State.  Miami had the best odds (54%) in the Coastal Division, but my simulation picked VA Tech to sneak by them.

What actually happened:  Clemson ran the table, and Pitt, despite only a 3% shot in the preseason, made the ACC Title game.

What went wrong:  Potential threats to Clemson (FSU and Louisville) were WAY worse than expected, so the Tiger's cruised.  VA Tech and Miami were also over-rated in the preseason, and Pitt also got a little luck with some close wins.

Big 12

2018 projection:  Oklahoma had the best odds to win the Big 12 (43%), but a tough road schedule.  My simulation picked the Sooner to finish 3rd with losses to TCU, Texas, and Oklahoma State.  Meanwhile, my math liked TCU.

What actually happened:  Oklahoma beat everyone except Texas, but then avenged the loss to the Longhorns in the Big 12 Title game.

What went wrong:  (This may sound like a broken record...) TCU and Oklahoma State were both not as good as expected, and the Sooner squeaked by the one team in the Big 12 that was under-rated: West Virginia.

Pac 12

2018 projection:  Washington had the best preseason odds (61%) in the North, but my simulation picked Oregon to win a 3-way tiebreaker with the Huskers and Stanford. Down south, USC (60%) had the best odds, while my simulation liked Utah.

What actually happened:  Washington won the North in a tie-breaker with Wazzou, while Utah won the South, despite odds of only 17%

What went wrong:  USC, Oregon, and Stanford were all over-rated, which allowed Washington and Utah to make it through.

My preseason analysis of the Group of Five was brief, but I also made predictions for all 10 of those division winners as well.  I won't go into detail, but here is a table the summarizes all of my picks and the actual winners:

(Note that I did not have the spreadsheet for the Sun Belt up and running in time for the preseason odds.)

When I look at all the data, I am honestly fairly encouraged.  The raw odds picked the Division winner 58% of the time and my simulation did almost as well (10/19 or 52%).  Moreover, if both the odds and my simulation agreed on the winner, I was right 7 out of 10 times or 70%.  In the 9 cases where the methods differed, the odds pick was right 4 times, the simulation was right 3 times, and neither was right only twice.

As for my final prediction of the NY6, last year at this time I had:

• Cotton Bowl: #1 Alabama vs. #4 MSU
• Orange Bowl: #2 Penn State vs, #3 Clemson
• Rose Bowl: Ohio State vs. Oregon
• Sugar Bowl:  Georgia vs. TCU 
• Peach Bowl:  Iowa vs. Miami 
• Fiesta Bowl:  Auburn vs. Boise State 

In reality, we had:

• Orange Bowl: #1 Alabama vs. #4 Oklahoma
• Cotton Bowl: #2 Clemson vs, #3 Notre Dame
• Rose Bowl: Ohio State vs. Washington
• Sugar Bowl:  Georgia vs. Texas 
• Peach Bowl:  Michigan vs. Florida
• Fiesta Bowl:  LSU vs. UCF

So.. that is only 4 correct out of 12, which is not great, especially since Alabama and Clemson were two of the four.  But, I did pick Clemson to beat Alabama in the Title game, so there's that.  Also, I feel like I was close on a lot of those teams, including picking TCU over Texas and Boise State over UCF, and I even commented how Notre Dame could easily go 11-1 instead of the 9-3 that I projected. They actually did one better that even my most optimistic prediction.

The other thing to note is that if I add up the odds for all the favorites to win their Division, I get an expected value of 10.2.  In the 17 races where I had preseason odds, the favorite won 10 times.  So, that is also very reassuring that my probabilities are reasonable.  The sample size is not large enough to do much more analysis than that, but the numbers look pretty good to me.  For example, there were a total of 64 teams with odds to win their Divisions between 0% and 5%.  Two teams (Pitt and Middle Tenn State) in that bin actually won, and 3/64 is 3.1%.   So, again my raw probabilities seem to be quite accurate.

So, overall, I think my projections did fairly well, and I think that we can expect a similar level of accuracy in 2019. For reference, here is a summary of my odds and simulation-based picks for 2019.

Just based on the sum of the odds, I should expect that the odds pick will be correct around 11 time out of the 19 total picks. I should also note that this year my simulation agrees with the odds prediction in 12 of the 19 cases, up from 10 last year.

I have one other topic when it comes to a reflection of last year. Once the season is over, it is possible to perform some additional analyses to see how a team's performance compared to the preseason prediction. In the preseason, I make a prediction about the expected value of wins for each team.  At the end of the season, each team will have actually won or lost all of those games. The difference between the preseason prediction and the actual result is due to a combination of three factors:

1. A team is either had more or less ability than I projected (i.e. their power ranking is higher or lower than estimated in the preseason)
2. A team's schedule is either harder or easier than projected, due to the changes in the ability of their opponents
3. Luck and/or execution.  For example, if a team has a 50% chance to win 4 of their games, you would expect them to win 2.  But, maybe they win 3 or 4.  Or, maybe they only win 1 or 0.  Is this due to good/bad luck or good/bad execution?  Either way, it is playing above or below a team's potential.

I figured out a way to mathematically separate these three factors, and it is possible to construct a waterfall plot for each team based on all these metrics.  Here is the plot for MSU last year:

So, based on my analysis, last year MSU won significantly fewer games than predicted because they were around 3.5 games worse than they were supposed to be (due to injuries, primarily).  MSU's schedule was actually just slightly easier than expected (the very small red bar).  All things being equal, MSU probably should have been 6-6 in the regular season.  But, MSU actually won one more game due to good luck / execution.  Of course, this analysis assumes that each team plays with the same average level of ability all year.  That is not true, but as an overall average, I think that this looks about right for the 2018 MSU squad.

It is a bit cumbersome to look at individual waterfall charts for all 130 teams, so I have a couple of ways to visualize the different categories.  First, I show the change in expected wins based on just the change in ability for each team.  There is a dot for all 130 FBS teams, but only the Power 5 teams are labeled

It is quite easy to see the teams that were both a lot better than expected (Kentucky, Virginia, Vanderbilt, etc.) and the teams that were a lot worse (Louisville, Florida State, and Virginia Tech... and MSU).  In addition, I have also plotted a comparison of the change in win total due to both the schedule (x-axis) and luck/execution (y-axis)

In this case, the teams in the upper right of the plot benefited from both a good schedule and luck. Notre Dame is a very clear outlier, which does make one wonder if they will actually be quite a good in 2019 as people think. Similarly, a team like Arkansas has both a tougher schedule than expected, and they were unlucky last year.  Might the Hogs be a bit under-rated this year?  Only time will tell.

That is all for now. Until next time, enjoy, and Go Green!