Luck & Skill in Modern Sports, Pt. 1
How do chance events affect the outcome of different professional sports?
Back in 2018 during my senior year of undergrad, I was fortunate enough to have an advisor in professor Peko Hosoi from MIT amenable enough to let me complete a mechanical engineering (ME) thesis on a slightly tangential topic from most traditional ME topics. I wanted to examine how luck and skill affected the outcomes of professional League of Legends (LoL) matches, and to compare the results with traditional sports like football, baseball, etc. Certainly not a typical slam dunk for a mechanical engineering thesis topic, but apparently it contained enough relevant statistical and mathematical analyses to fulfill the course requirements. If you want to judge for yourself, you can still read it in its poorly-scanned form on the MIT library website. I’m also happy to report it comes up on the 7th results page when you Google “MIT Cameron Douglas Luck and Skill League of Legends.”
Despite the research itself being 7 years old at this point, I still often think about the underlying mechanisms that affect the impact of luck and skill on the outcomes of sports games, and how insight on the topic can be leveraged more broadly. A deep understanding about what factors increase or decrease skill expression in sports can be used by everyone from coaches who want to find a competitive edge, to leagues looking to modify rules in order to create more competitive seasons, to sports bettors who want to optimize their betting portfolio, and even to investors considering an acquisition of a professional team or league.
The upstream and downstream effects of luck and skill in pro sports are far-reaching, but in the interest of sanity and appropriate article length, I’ve split this complex set of topics into two short-ish articles, for which this is Part 1 (as the title suggests). This article focuses on the underlying mechanics that influence luck and skill in sports and how to quantify the balance of luck and skill on the outcome of games. We’ll use this math to actually chart popular sports along the axis between pure luck and pure skill before coming back in Part 2 to look at the implications of this information. Of course, understanding what types of sports are more determined by luck and which are determined by skill will be useful in and of itself, but I am also keen to unpack how the underlying factors leading to these outcomes can be manipulated to achieve different ends.
If, after digesting these two articles, you are interested in learning more about the topic, I highly recommend reading Michael Mauboussin’s The Success Equation that served as the foundation for my initial thesis work, reaching out to me, or even giving my thesis a once-over if you have tolerance for rough grammatical errors! In the meantime, if you need a quick warm-up to get some extreme examples of how low-probability events can affect sporting events, enjoy this wild sports compilation:
Foundations for Luck & Skill in Sports
The first thing we need to do before proceeding down a conceptually difficult path of analysis is to start with our definitions for the variables we want to study. We need definitions for luck and skill within the context of sports:
Luck: The confluence of unpredictable, uncontrollable, and/or random factors that affect the outcomes of sporting events, independent of a team’s or player’s inherent ability, preparation, or strategy.
Skill: The totality of an individual’s or team’s ability to affect the outcomes of sporting events. The consistent, repeatable application of athletics, tactics, preparation, decision-making, and other improvable factors that increase the probability of winning over time.
Framed differently, luck is everything that affects the outcomes of sporting events once the average skill of the players and teams facing off has been factored in. Luck is the head cold that negatively affects a star player for one night, or the sun beam that blinds the QB during a game-winning drive, or the gust of wind that blew a golf shot from the fairway into a sand trap, but it’s not just acts of god that determine its impact on a sports league, it’s also the rules of the game, the scoring system, and even structural league governance decisions like number of games and match format. A colloquial saying that goes something like “luck is the culmination of chance and preparation,” but for our purposes throughout these articles, this is definitively not the case. In the definitions above, you can think about chance and luck being effectively interchangeable.
As I’ve intimated in the early parts of this article, the balance of luck and skill in different sports is different, and it’s important to note that even for a given sport, the balance of luck and skill is continually changing, even if the underlying rules and season structure don’t change. It’s a topic for another day, but luck and skill have an interesting interplay where luck becomes a much more important factor in outcomes as the average skill level rises in a field, and the distribution of skill within condenses— two effects that usually go hand-in-hand:
“The key is this idea called the ‘paradox of skill.’ As people become better at an activity, the difference between the best and the average and the best and the worst becomes much narrower. As people become more skillful, luck becomes more important. That’s precisely what happens in the world of investing.”
In all endeavors, there are elements of skill and elements of luck that govern outcomes, but what factors control the delicate balance of luck and skill expression in the outcome of sporting events specifically? Beyond individual instances of extremely good or bad luck that were highlighted in the compilation above, a huge host of structural, environmental, and even some player-pool-based factors affect the balance of luck and skill in all sports. I’ve listed as many individual factors that affect the amount of luck and skill that affect sporting events as I could conceive in the table below, and it’s worth mentioning that even for “Luck-heavy” embodiments and bracketed example sports below, player and team skill is the major governor of outcomes.
On the surface this table appears relatively straightforward and simple, but I definitely recommend spending some time to consider each factor in the context of multiple sports. The implications of each factor seem to never end, and it’s extremely fun to think about each one in the context of a single sport!
Even with just this descriptive list, rough weights of influence, and a smattering of individual example sports, you can already probably already start to guess what types of sports and which leagues are more skill-dominant and which have luck impact the outcomes of matches more regularly. For the lack of any ambiguity, here’s my high fly-over of the major professional sports leagues in the US, and the English Premier League (EPL) as a standard soccer league example based on the info above. Notably, all these leagues play best-of-one series during the regular season:
MLB (Baseball): Mix of luck and skill. A mixed bag of luck-heavy and skill-heavy elements. On the luck side, varying conditions/stadiums, high number of players per team with limited opportunities for star pitchers to play due to injury risk, and a relative narrow skill band overall. However, high-skill elements like a high, consistent number of scoring chances per game do enable skill expression.
NBA (Basketball): Heavily skill-oriented. Very skill-heavy predominantly due to very high number of scoring opportunities per game, consistency in playing surface, generally injury-light games, and small team size. Penalty systems are pretty low impact overall, but do add some luck.
NFL (Football): Mix of luck and skill. Very similar to baseball, but perhaps slightly more skill-oriented due to incremental scoring with field goals, and the fact that the most important position on the field, the QB, has an outsized impact on the whole game and typically every game.
NHL (Hockey): Luck-heavy. Very luck-heavy in the grand scheme, given the low number of scoring opportunities per game, large team size, and very punishing penalty and tie-resolving mechanisms. Ice rinks are quite consistent and substitutions are freely allowed, but these features aren’t enough to overcome the luck-heavy elements of the sport.
EPL (Soccer): Luck-heavy. Very similar to hockey, but maybe even more luck-oriented given that games tend to have even fewer goals scored. However, the variance in player skill within a league may be higher in the EPL than in the NHL, and the penalty system in soccer is not as dramatic.
With this qualitative conviction about what sports and what types of leagues are more skill-based than others, the question becomes: how do we prove this with math?
Quantifying Luck & Skill in Sports
In short, the two predominant mathematical approaches to discerning the balance of luck and skill in a sport with distinct seasons are a “Persistence” approach and a Bayesian approach. There are additional methods for determining luck and skill, including an autocorrelation approach for tournament results that can be applied to other sports, but I haven’t included these analyses here— perhaps that will be an activity for a future article! I will provide a quick, over-simplified summary of each main approach here before we use the Persistence method to chart the sport leagues listed above, but if you are curious about learning more math in this realm, Mauboussin’s book is a great place to look.
I think the Persistence approach is an incredibly clever statistical method (which I can say because I didn’t create it). It is based on the simple insight individual skill should be persistent throughout the entirety of a season of sport, so team’s records should be similar or identical in the first half and second half. To actually condense this insight into math, you start by plotting the first-half win % for every team in the league during the season against the second-half win % (over multiple seasons if you have the data). You end up with charts that look like those below from my thesis, where each dot represents the record achieved by a team over the course of ~5 seasons of League of Legends— larger dots means more teams achieved this record.
Once you have this data organized and visualized, you can leverage the second key insight, which is that for a game of complete chance, every single team should always have a record of 0.500 in the first half and the second half of the year with a big enough sample size. Every team should be clustered at the center of the chart. On the other hand, if a game is purely skill-based, every team should achieve the same record in the first and second half of the season, but every team will end up with a different record. The best team wins every game in both halves of the season, the second best team wins every game except against the best team, and so on, all the way down to the worst team losing all games in both halves of the season— so you end up with a line from the bottom left to the top right of the chart.
In reality, every sport has some balance of luck and skill, so you end up with distributions that look like those above— where I’ve plotted League of Legends team records for best-of-one formats on the left (more luck-based) , and best-of-three formats on the right (more skill-based) for ~5 seasons. To convert the clear visual into a single value that quantifies luck and skill, you divide the variance of each distribution along the diagonals of the graph— where a value closer to 1.0 indicates a completely luck-based game (i.e. the distribution is equal in both diagonals), and a value closer to 0.0 is more skill-oriented (i.e. the distribution is more linear bottom-left to top-right like you see in the best-of-three chart). You then subtract this number from 1.0 to produce a spectrum of luck and skill!
The second approach, which uses Bayesian statistics, is slightly more straightforward. In the most oversimplified terms, you effectively start with the prior assertion that a given sport is 50% skill-based, p(skill), as a starting point— for reference p(luck) is just 1-p(skill). You then update the prior iteratively by feeding individual win-loss records into the equation below one after another. Every win-loss record increases or decreases the prior, p(skill), and for a game where skill out-powers luck to determine game outcomes, eventually p(skill) converges to 1.0, indicating a 100% chance that the game is determined predominantly by skill.
To actually quantify the comparative balance of luck and skill, you run 100’s or 1,000’s of iterative simulations by plugging in team’s records in random orders into the iterative process above. You can then measure the average number of win-loss records that it takes to converge to 1.0— affirming the prior that a game is skill-based. The two charts below visualize this overall process for 500 simulations of League of Legends records, again for best-of-one formats on the left, and best-of-three formats on the right. It’s difficult to see with so many lines running across the charts, but the best-of three format converges to 1.0 at an average of ~0.5 win/loss records faster than the best-of-one format, indicating a higher impact of skill on outcomes (which is to be expected).
With some quantitative reasoning in hand, we can finally compare the various professional sports leagues mentioned above in terms of their balance of luck and skill! As mentioned above, we will do this using the Persistence method, which has been well-studied for most major sport leagues. Charting these the regular season of these leagues on a spectrum from pure luck to pure skill, and including some additional activities like flipping coins, picking stocks, and professional cycling, provides the following breakdown:
You could probably spend a lifetime looking at the table of factors above and thinking through the placement of each sport on this spectrum, but it’s great to see that our qualitative assessment was roughly correct! It really is fascinating how much more skill-based sports like NBA basketball and professional cycling (which is a good proxy for endurance sports more broadly) than sports like baseball and hockey when they are played in the regular season in a best-of-one format. It makes sense that endurance sports exhibit the largest skill expression, since in general if you are faster or have better endurance than other athletes, it’s mainly low-probability injuries or freak events that could dislodge the best cyclist or best marathon runner from winning a race if they are all in peak form. In other words, major upsets in endurance sports are rare for a reason!
I think the fact that outcomes in NFL games are more skill-based than a sport like hockey also speaks to the football’s condensation of outcomes into a key role with the quarterback. The same is definitely true for pitchers in the MLB, but pitchers aren’t playing every game in order to prevent major injury. This thread of specialized roles does not translate into a very high skill expression for roles like hockey and soccer goalies though, which might speak to the fact that their performance is much more intertwined with the players around them playing defense. It really is a fascinating set of minute effects to analyze.
More broadly, though, it’s somewhat of a relief to see that the outcomes of all professional major sports matches are determined predominantly by skill. Of course, luck can still play an enormous role in individual games that enable a worse team to beat a better team. However, I think verifying the belief that even as average skill level has risen and condensed for professional sports leagues, individual and team skill still determines the majority, or vast majority, of outcomes is useful— it certainly wasn’t a given. It will be fascinating to think through how this spectrum and the balance of luck and skill might change as we start to have average skill condense, and luck begins to play a bigger factor in line with the paradox of luck and skill so eloquently stated by Michael Mauboussin!
But anyway, I’ll save more analysis and conjecture for next time when we dive into a broader look at what the implications of all this knowledge might be. Now that we have an understanding of what causes luck and skill expression in sports, and where each sport falls on that spectrum, there’s lots more to dig into!