How Much Do Recruiting Rankings Matter?

How much do recruiting ranking really matter? A few weeks ago, I decided to investigate this debate in more detail.  I have started to assemble a database of players from all 14 Big Ten schools using each and every committed recruit back to 2007.  For the source of my data, I am using the Rivals database, as I have easy access to it and the format made the data easy import.  My goal was to try to categorize and quantify the overall success of each player in order to visualize and analyze the true value of the initial Rivals recruiting rating.  So far, I have inputted all the players in the MSU and UofM databases, and the result so far are quite interesting. For those that may not be familiar with the Rivals ranking/rating system, they use a 5-star system, but they also use a “rating” system that is broken down into smaller intervals (4.9 to 6.1, for whatever reason).  The table below shows how the star-ranking compares to the simple rating as well as give the number of players in each category (based on the 2017 class).  I should be noted that the 4.9 – 5.1 ratings seemed to fall out of favor at Rivals after 2009 or so, and those ratings now seem to have been lumped into the 2-star (5.2 to 5.4) categories.

In order to be consistent and quantitative, I had to create a set of rules to measure the overall productivity of each player.  These rules are certainly debatable, but I think they are reasonably fair.  First of all, as I mentioned above, I decided that I would only consider players that are listed in the Rivals recruiting database that originally committed to the school in question.  That means that in general, walk-ons and transfers are not considered.  However, I decided that I would add walks-ons to my database if they made an All-Big Ten team or made the NFL (for example, Jack Conklin).  But, a player like Russel Wilson would not be included for Wisconsin (as his productivity would mostly be credited to NC State, the school that recruited him) and Jake Ruddock would still be counted as an Iowa Hawkeye.

As for quantification, I decided to set up a scoring system based on five factors:  number of varsity letters, number of starts, career highest position on an All-Big Ten team, NFL draft round, and years of play in the NFL.  My idea is that these factors build on each other.  The basic minimum to be a productive player is to earn a letter.  So, I award half a point for each letter earned, up to 4 letters.  The next level up is to be a consistent starter, so I award half a point for every 12 career starts (or more precisely, “# of starts”/24 points).  The next level up from that is perform at an All-Big Ten level at some point.  So, I award 2/x points, where x is the All-Big Ten Team (i.e. 1^st team is x=1, 2^nd team is x=2, 3^rd team is x=3, and honorable mention is x=4.)  I use a similar system for the NFL draft, only here I use 3/x, where x is the round of the draft (so a 1^st rounder gets 3 points, while a 4^th rounder gets 0.75 points).  Finally, I also award 1 point for each year played in the NFL.  By “played” each player must have generated at least some stat during the regular season.  Basically, if NFL.com showed a player as active in a given year, I count it.  I then sum up the points for each category to get a “player productivity score” for each player, which generally goes from zero (for a total non-contributor, AKA, a “bust”) to over 10 for a player like Darqueze Dennard (current NFL players, 1^st round pick, All-Big Ten 1^st team with 40 starts and 4 letters).  I also made the final rule that I would not compute a score for a player until they leave the program.  That said, older players have more time to accumulate NFL seasons, so there is a bias towards players from farther back in time.

While I like this complete quantification, I thought that it might be more instructive to take a slightly simpler approach.  The five factors are intended to essentially separate players into 5 basic categories: NFL players, All Big-Ten performers, Starters, Contributors, and Busts.  So, I decided that perhaps I should just use those 5 categories.  So, I basically group each player into one of those 5 categories using the following metrics.  If a player is good enough to get drafted OR actually play in a real game in the NFL, I consider them to be an NFL player.  If the player made any All-Big Ten team (even just honorable mention once) they are in the All-Big Ten category (assuming they are not already in the NFL category).  If the player started in at least 7 games (more than half a season) I consider that good enough to have been a consistent Starter.  If the player earned at least 3 letters, I group then as a Contributor, and otherwise, they are a Bust.  This system is not perfect, and it does not perfectly match the “player productivity score” but I think it is fair.  Perhaps more importantly, it is a framework that can be easily applied across the board to all teams and hundreds of players.  That is where the real power lies: with the ability to consider all players and not just cherry pick the Le’Veon Bells and William Campbells of the world.

Now that the methodology is clear, what does the data show?  In the following charts, I present the breakdown of MSU and UofM players by category as a function of the Rivals rating for the 2007-2012 recruiting class (128 MSU recruits/players and 141 UofM recruits/players).  I cut the data off at 2012 because all players from that class have completed their eligibility.  If I include the 2013 class, this tends to bias the data towards NFL players (who leave early) and busts (who leave early for other reasons).  Also, I decided to lump together the “6.1” and “6.0” category into just one category called “Top 75” as these categories are small to start with (MSU has only 3 players in this category and Michigan has 9 for this timeframe).  For the main bar charts, I also lumped together all the Rival’s ratings below 5.4, which general includes walk-ons (which I assign to be “4.9”), specialists, and a small number of other players who are obviously under the radar.  Following each bar chart is a similar breakdown showing the percentages instead of the raw number of players.  In this case, I exclude the “<5.4” category all together, as I don’t include all the walk-ons on a given team and specialists and dudes like Jack Conklin are just going to mess up the statistics anyway.

OK, enough of my yapping, here is the data for MSU:

If you are anything like me, this data is extremely surprising.  I am certainly not a “stargazer,” but I expected to see a least a weak trend suggesting that the higher rated players were generally more productive.  Honestly, that it not what I see here.  For the MSU data, it is true that 66% of MSU’s Top 75 recruits went on to the NFL.  But, here is the problem: for 2007-2012, that is only 3 total players (Gholston, Rock Baker, and bust David Barrent).  However, there is reason to believe that the percentages are about right.  I did manage to input just enough data for Ohio State to capture the productivity of the Buckeye’s Top 75 recruits from 2007-2012, and the results suggest that two-thirds of the Buckeye recruits were All-Big Ten or NFL level players (with the majority, 50%, being NFL players).  So, this does suggest that getting those 6.1 and 6.0 rated high level players does improve the odds of getting high productivity players.  However, the vast majority of MSU’s recruits in this time period were rated between 5.9 and 5.6 and the percentage chart shows very little difference in the distribution of productivity for these four categories.  Basically, two-thirds of the players in all four categories were at least starters, 40-50% were at least All-Big Ten level, and 15-20% were NFL caliber.  By the time the rating gets down to 5.5 or below, there is a clear drop in players who are good enough to start (to below 50%).  Yet, there are clearly still several NFL and All-Big Ten players from even these lowly categories.

For comparison, here is the data for Michigan:

While the data certainly is different, there are also a lot of similarities to the MSU data.  First, it should be obvious that UofM’s average Rivals rating is higher than MSU’s.  But, what does it get them, besides 3^rd place finishes? For one thing, the bust / simple contributor rate is way higher than it is for MSU. It is over 45% for all categories down to 5.4.  After that, the distribution of high-level contributors is pretty flat.  The Wolverines get NFL or All-Big Ten level performance out of about 30% of their recruits for the Top 75 to 5.8 ratings and closer to 20-25% for the less productive (for Michigan) 5.7 and 5.6 rating categories.  Similar to MSU’s situation, the 5.5 rating category shows a dip in productivity.  For Michigan, you might be tempted to point at the apparent spike in productivity for the 5.4 category, but this is literally 3 guys.  A quick look back at the MSU data clearly shows that UofM’s rates are noticeably lower.  MSU seems to be having more success developing players across the board, regardless of rating.  Also notably, Michigan seems to do far worse with its Top 75-level talent than either MSU or OSU.  It will be interesting to see how the rest of the Big Ten fair on this metric. 

Now, I am sure that Michigan fans will claim that this is all due to RichRod and Hoke, who were objectively terrible, and now that Harbaugh is in town the Wolverines will suddenly be pumping out NFL and All-Conference players across the roster. Sure. Maybe.  The Hoke-recruited and Harbaugh coached 2012 class did very well, with over 50% of the players being All-Big Ten level or better, but so far the (higher ranked) 2013 class does not look like it is going to equal the productivity of that 2012 class.  Time will tell, I suppose.

Now this is just a quick interpretation.  There is a LOT to unpack in this data.  But, to me there are 3 main conclusions to draw:

1) Getting Top 75 level talent (5-star and high 4-star) helps.  Based on the small data set, these players are more productive, on average.  Basically, the odds are better that they will be NFL/all-Conference level players.  This make sense, as these are the caliber of players that teams like Alabama, Clemson, and OSU stock up on seemingly every year.  But, for every Big Ten team not named OSU, these players make up less than 10% of the total recruits.  For teams not named Michigan or OSU, it is less than 5%.

2) For the 5.6 to 5.9 range of players, the performance seems to be surprisingly consistent across the entire range.  But, better coaching seems to be able to draw more talent out of this group.  Considering this group makes up the majority of recruits for all of the recently and historical competitive Big Ten teams (OSU, MSU, Wisconsin, Penn State, Michigan, and Nebraska), this suggests that the impact of good coaching is actually really, really important, perhaps even more than we thought.

3) While the likelihood of a bust or a minimal contributor shoots up for players rated 5.5 or below, there certainly is talent here.  In fact, my preliminary data back to 2007 suggests that roughly a third of all Big Ten players that make the NFL start off rated 5.5 and below.  Once again, good coaching is likely a factor here.

As a final thought, I spent a long paragraph at the beginning talking about the more quantitative “player productivity score,” which I originally thought would be more useful in this analysis.  Well, the reason I partially abandoned that idea is due to the follow plots of the productivity score vs. Rivals rating for both MSU and UofM.  As a warning, star-gazers might want to avert your eyes.

Based on this analysis, there is essentially no mathematical correlation (i.e. R2 ~ 0) between player productivity and recruiting rating.  Now, for some of the reasons I mention above (higher productivity of Top 75 players who don’t attend Michigan and the lower productivity of players rated 5.5 or less) I don’t think that this is the whole story.  But, it is a striking, and for me, a very surprising result.

That is all for now, but I do not consider this the end of the story.  I plan to continue to build the database and analyze the data.  When I find something interesting, I will continue to post it here.  I think that this is going to be fun.