For close to a decade, I have been utilizing a set of mathematical tools that allow me to better understand college sports. One of the tools that I have developed over the years is a method to simulate the results of the full college football season, including accurate odds for various future events such as division and conference winners and regular season win totals.
Every year, a key input into that simulation is a consensus ranking of all FBS-level teams. A second key input is the uncertainty (i.e. historical accuracy) of those rankings, which I summarized in the first installment of this preseason series.
Today, I wanted to provide a more specific example of the type of information that my simulation provides and how it performs historically. The most recent set of data is that from last year. So today. let's look at the predictions that my simulations made about a year ago regarding the 2022 season and how those predictions turned out.
2022 Division and Conference Prediction Results
Before we jump directly into the numbers, I would like to provide a quick primer on my simulation methodology. I have developed my own power ranking system which is partially based on the preseason rankings early in the season. I use these power rankings to derive point spreads for all future contests for the entire season, including potential playoff games.
Point spreads correlate to win probabilities, and if one can predict the odds of a set of future events (football games) the full system (the entire college football season) can be simulated using what is known as a Monte Carlo simulation. All that means is that a computer can effectively roll an extremely substantial number of digital dice and the result is a collection of odds for how that football season will play out.
I also conduct a slightly different type of simulation, which I call my "disruptive" simulation, where I first assume that all the projected favorites will win every game. I then adjust that result to inject a historically accurate number of road upsets. This simulation will often give the same result as the Monte Carlo simulation, but at times there are some interesting differences.
With that said, Figures 1 and 2 below provide a summary of my computer's 2022 predictions for all the FBS division and conference champions and the actual results. The correct predictions are highlighted in yellow. I also included the preseason odds for each of these events.
 |
| Figure 1: 2022 preseason division winner projections with odds and the actual results |
 |
Figure 2: 2022 preseason conference winner projections with odds and the actual results. |
For the divisional races shown in Figure 1, only six of the 16 preseason divisional favorites (based on the preseason rankings) ended up winning their respective divisions. For the conference races, only five of the 10 projected winners ended up winning the conference championship game.
While this success rate may seem low, it is not, at least not based on the odds that I calculate. The third column in both figures gives the preseason odds for each of the favorites to win their division or conference. The sum of these odds gives the "expected value" for the number of predictions that is likely to be correct.
The total expected number of correct picks in the two combined figures is 10.9. The actual number of correct predictions was eleven. In other words, the math checks out.
As for my disruptive picks in 2022, this method did successfully pick Michigan as both the winner of the Big Ten East and the Big Ten as a whole. However, it also incorrected suggested that Boise State would win the Mountain West. The only other pick that was different using my disruptive method was a pick of Marshall over Appalachian State in the Sun Belt East. Coastal Carolina wound up winning that division.
As a final note on this data set, the last column of both figures gives the preseason odds for each of the actual division and conferences winners. The thing to note here is that the teams that were considered an extreme longshot in the summer (such as TCU and Tulane) still had divisional odds that were in the range of 5-10% in my system.
While these numbers are not large, they are larger than the odds that you will see on other sites for similar longshots. Systems such as ESPN's Football Power Index (FPI) will typically give odds of less than one percent for any team outside of the preseason top four or five in a conference. This is because models such as the FPI do not factor in the uncertainty in the preseason rankings. My model does.
2022 Preseason Top 25
To complete this retrospective analysis of the 2022 season, Table 1 below shows the teams listed in the final AP top 25 poll. For reference, I have also included Phil Steele's preseason ranking and I color coded each row depending on whether a team did a lot better than expected (in green), finished at a ranking close to the preseason ranking (yellow), or underachieved (in orange or red).
 |
| Table 1: 2022 Final AP poll with a reference to the preseason Phil Steele rankings and an analysis of the preseason and postseason expected win total. |
The color coding provides a quick snapshot of the accuracy of the 2022 preseason rankings. I only count a total of nine teams that finished the season roughly where they started, although that includes four of the teams in the final top five (Georgia, Michigan, Ohio State, and Alabama).
I count eight teams who underachieved slightly, including two preseason top 10 teams which dropped over 10 slots (Clemson and Notre Dame), two preseason top 20 teams which just slipped out of the top 25 (North Carolina State and Iowa), and four preseason top 25 teams who did not wind up ranked (Auburn, Oklahoma State, Baylor, and Louisville).
There are also five teams which Phil Steele had ranked in the top 15, but which wound up unranked (Texas A&M, Miami, Wisconsin, BYU, and Oklahoma).
The remaining 14 teams in the table all overachieved significantly compared to the preseason ranking. The most upwardly mobile teams in 2022 were TCU (ranked No. 51 in the preseason and finished No. 2), Washington (No. 60 to No. 8), Tulane (No. 65 to No. 9), and Troy State (No. 92 to No. 19).
When I perform this analysis next summer, I would expect similar trends. Keep that in mind when looking at the 2023 preseason rankings.
Ability, Schedule, and Luck
Finally, the right side of Table 1 contains some analysis that helps to explain why some of the teams moved up or down in the rankings. I also included Michigan State at the bottom of the table for the purposes of this analysis.
In this case I used my preseason and postseason power rankings to calculate the expected number of regular season wins for each team both before and after the season. I was then able to apply a few mathematical tricks to separate out the impact of the three general reasons why teams wind up being better (or worse) than expected: ability, schedule, and luck.
"Luck" as I define it is the ability of a team to win more (or fewer) games than the point spreads suggest. For example, if a team were to play 12 games where the spread in each game was a pick 'em that team would most likely finish 6-6 on the season. If that team were to win eight games (for example), they would have +2.00 games of luck.
I calculate changes in wins based on schedule using my strength of schedule calculation, which is the expected number of wins a representative top 25 team would have with any given schedule. Since I already measure this factor in terms of wins, it is easy to calculate the difference in schedule strength before and after the season.
I attribute the remaining changes in expected wins between the preseason and postseason to changes in the actual ability of each team. That is, the team is either better or worse than expected in the summer.
Michigan State overall won almost 3.5 fewer games than expected in 2022. The Spartans' luck was negligible, but they lost a full game in expected wins due to a harder than expected schedule. This is reasonable considering that Washington, Penn State, and Illinois were all significantly better than the preseason publications suggested. That said, Michigan State underachieved primarily because the team was simply not as good (ability wise) as expected by 2.5 games.
A glance at the data in Table 1 and a look at the data for all FBS teams (not shown) provides a good rule of thumb as to how much each of the three factors explained above impacted a team's performance relative to preseason expectations.
Changes in strength of schedule have the smallest impact. On average, an easiest or harder schedule is only worth about half of a game over a 12 game schedule. Luck is the second most important factor, as it tends to impact a team's final win count by an average of plus-or-minus one game.
But the biggest factor that moves a team's win total up or down relative to preseason expectations is ability. That is worth roughly two games, on average, up or down. The biggest unknown in the preseason is exactly how good each team is going to be.
These rules of thumbs are just averages, however, and in any given year some teams will be especially strong (or weak) in any of those three categories. Table 2 below summarizes the top 10 and bottom 10 teams from the 2022 season in terms of changes in ability, schedule, and luck as well as the team's overall change in regular season wins compared to the preseason.
 |
| Table 2: Top 10 and bottom 10 teams in 2022 in the change in the number of wins or expected wins overall and based on luck, changes in ability, and changes in schedule strength. |
On the extreme ends of the spectrum, changes in the preseason understanding of each teams' ability resulted in a swing of upwards of almost +5.0 wins for James Madison and downwards to almost -6.0 wins for Miami. The luckiest team in FBS last year was Coastal Carolina (+3.5) while Memphis (-2.5) was the least lucky. As for changes in schedule strength, Virginia Tech (+1.5) got the biggest break while Purdue (-1.4) had the toughest schedule upgrade in the nation.
A closer look at Table 1 and 2 provides more insight into what went right or wrong for each team in 2022. For example, Miami was the biggest disappointment in 2022 and the reason for this appears to be purely due to a drastically lower level of ability compared to what was expected in the preseason. The Hurricanes had a slightly easier schedule than expected and some luck in 2022. To a lesser extent, BYU had a similar profile.
Texas A&M, Wisconsin, and Oklahoma all underachieved by a similar amount in 2022 (roughly four games) but they got to that number in different ways. Texas A&M has the biggest gap in predicted ability. The Aggies also were quite unlucky (-2.0), but this was partially offset by an easier than expected schedule (+1.2).
Wisconsin underachieved in 2022 due to a lack of ability and some bad luck but had a neutral schedule. Oklahoma was negative in all three areas which suggests that the Sooners dip in ability was smaller than the dip at Texas A&M and Wisconsin.
Finally, it is interesting to look at the teams whose final win total was bolstered the most by luck last year. TCU won 2.5 more games than expected due to luck, but they were also a lot better than expected (+4.7 games in ability). Other teams that had a notably high luck metric in 2022 include USC, Troy, South Carolina, and Oklahoma State. This suggests that the final record for these teams last year might be a bit misleading and which might translate to these teams being overrated coming into the 2023 season.
It should also be noted that of the teams that finished 2022 in the top 10, the second most lucky team (behind TCU) was the Michigan Wolverines with +1.5 games of luck. While Michigan was a little better than expected in 2022, luck was the biggest single contributor to their 12 regular season wins relative to preseason expectation of just over 10 wins.
Now that we have a more detailed understanding of what happened in 2022, it is time to shift our focus to 2023 and a thorough analysis of the season to come. Stay tuned.
Comments
Post a Comment