Game Length and Completion Rate: Is Shorter Really Sweeter?comments powered by Disqus
Posted on Tuesday, October 15 2013 @ 09:35:48 PST
This member blog post was promoted to the GameRevolution homepage.It is a statement that carries the inevitability of climate change in Winterfell: “Games are getting shorter.” Despite a startling lack of popular data-driven analysis on the topic, you would be hard-pressed to find a gamer that disagrees. Shorter games are just fine, as the argument goes, because with work, family, and a myriad of other entertainment options available, the average person does not have time to complete longer games in the first place.
But why theorize when all the information we need to address the question is publically available? Are longer games less likely to be completed than shorter games? Join me on a data adventure!
Declining single-player game length has been analytically argued with modest frequency throughout the seventh console generation, but never more than when this article from Ars Technica surfaced with quotes from developers, whom cited two main reasons they anticipated a decline in average game length:
The article commented on the graph data, saying, “According to achievement tracking service Raptr, only 7 percent of registered users on the Xbox 360 version of Red Dead Redemption finished the last campaign mission.”
Fair enough. But Mass Effect 2 was certainly no slouch, time-wise, and that game has the highest completion rate out of any on the chart. And while Raptr’s user base is sizeable, it might not be representative of the average gamer from title to title. To properly compare completion rate and game length, we need global completion statistics (or at least a representative sample) and an estimate of not just average hours played—the other graph in the Ars Technica article—but average game completion time.
Conveniently, both completion rate and completion time are freely available online, albeit with some limitations. Steam offers access to global achievement stats for any game that has Steam achievements, and the fantastic website HowLongToBeat provides average user-submitted completion times, with a special statistic for main campaign completion.
Unfortunately, as with any data source, there are caveats. Using Steam achievements limits us to PC-only or multiplatform games—and to be truly fair, we can only use multiplatform games that released simultaneously on PC and consoles, lest we find ourselves comparing PC-only gamers for one title against multiplatform owners for another. We lose all recent EA games, which are not available on Steam, and we lose all Ubisoft games, which do not appear to share their achievement data with Steam.
Furthermore, to keep our player base and ‘type’ of game as consistent as possible, we limit ourselves to only high-profile retail releases with a substantial single-player component, cutting out all indies (and feeling the pain of the missing EA and Ubisoft data). Because we are tracking game progress through achievements, we ALSO lose any game that does not have a single achievement for game completion on any ending at normal difficulty or lower. Sorry, Saints Row: The Third.
Following all of the data cleanses, we are left with a sample of 37 games over the past few years. Not nearly as much data as I’d like in order to draw a generalized conclusion, but when does one ever have enough data in their life, am I right?
In speaking of data, let’s hop to it:
At first glance, it looks like the Ars Technica article was spot-on. These games have an average completion rate of 38%, and there appears to be a cloudy but present negative exponential relationship between completion time and completion rate. In fact, let’s go ahead and draw that exponential line of best fit into the chart:
Done! Longer games get completed less, the end.
Buuuuuut… If we take a second look, that data is very cloudy, indeed. The R-square for the line we just drew is only 32%, meaning that 68% of the variability in completion rate from game to game cannot be explained by the average completion times for each game. In fact, if we were to cluster the data into quadrants, we would have groups of completion rate data with almost ZERO relationship to completion time:
Well, that puts things into a different light! We have three distinct clusters with no relationship to completion time, and a fourth that does appear to be linear. We can see a sharp overall drop in completion rate right around the 15-hour mark, which leads me to believe that completion time is a factor in completion rate mainly at larger magnitudes—all else being equal, a 25-hour game is nearly half as likely to be completed as a 10-hour game. But surprisingly, the difference between 6 hours and 12 hours is almost non-existent, despite that maximum being double the minimum. It seems that, in the same way that moviegoers might consider any movie under 2 hours to be ‘normal’ and any movie over 2 hours to be ‘long’, gamers also have very stark completion breakpoints in single-player campaign length.
But how do we explain the differences in completion rate within each time band? In both the 6-8 hour and the 10-12 hour bands, we have games pushing 60% completion and games below 40% completion. If campaign length differences aren’t driving that variance, what is?
Why, the quality of your gaming experience, of course! Let’s add the Metacritic score of each game to the graph:
Happily, we see that consumers will not, in fact, wantonly devour whatever product is shoveled beneath their noses: on average, critically successful games have higher completion rates. Of course, much as with completion time, Metascore does not necessarily have a strong linear relationship with completion rate. Rather, completion likelihood can again be bucketed into a binary ‘high’ and ‘low’ probability—the 85 Metascore standard adopted by the game industry seems to be a fine quality benchmark for this break point.
To summarize thus far, single-player game completion rate is dependent on both campaign length and game quality, but the relationships are based not on continuous linearity, but binary breakpoints: around 15 hours length and 85 Metacritic score. What explains the variability within these quadrants? We don’t know, but it doesn’t seem to be campaign length, and it doesn’t seem to be overall game quality.
In its closing paragraphs, the Ars Technica article makes a second argument about shrinking game length: “If only ten percent of a game's player base sees a game to the end, and 90 percent makes it over 60 percent of the way there, of course games are going to keep getting smaller—it's a matter of efficiency for developers.”
I am fairly confident that the author pulled that statistic straight out of thin air, but the question is valid. We know our average game completion is 38%, but what the heck are the other 62% of players doing? Do they play for five minutes and quit, or are they slogging through 60% of a campaign before losing interest?
We can answer this question by further analyzing Steam achievements. Many games have not only a campaign completion achievement, but campaign progress achievements at fixed intervals throughout the game. As a gamer, I hate these achievements because they rip me out of the story, but as a fake statistician, I love them because I can infer that anyone who has completed a later progress achievement has also completed the prior ones. We can map the % of campaign completion to each achievement based on where in the campaign it is awarded and then chart the percentage of players who have reached each milestone. This will effectively show us the player drop-off over the course of a game’s content:
(Note: 14 games dropped out of our sample due to the lack of progress achievements)
How incredible is that? The steep decline doesn’t occur after 60% of a game is completed—it occurs after only 5-10% completion! Nearly everyone makes it through this first 5-10% of content, there is a massive drop-off between 10% and 40%, and then past 40% there is a shallow, linear decline all the way down to the campaign completion.
Two things strike me:
1) From a data perspective, it is shocking how players all follow an identical relative distribution curve, despite differences in campaign length and quality from game to game—giant drop off up front, slow decay in the back. Now THAT is a negative exponential trend we can believe in.
2) From a game development and publishing perspective, this data suggests that it is most prudent to front-load with awesome set pieces and gripping story beats, rather than pushing development work into an ‘epic’ finale at expense of earlier content. This data isn’t only arguing for shrinking game length, it is arguing that the best content a game has to offer should be available immediately to all players and not hidden away as a ‘reward’ for clearing earlier content.
Of course, there is a fundamental assumption driving all of this analysis: gamers aren’t basing purchasing decisions on campaign length. If gamers who would never complete a 12-hour campaign don’t purchase 12-hour long games, then the completion rates of those games are biased upwards and are not representative of the negative effect of a longer campaign.
As mentioned earlier, I attempted to control for this by only including high-profile releases in my sample. But in the interest of full disclosure, I’ll let you all decide how heavily self-selection has biased our data—here is the original data chart with all games revealed:
With regards to completion rate, we’ve seen that campaign length matters broadly, as does quality. But all games, long and short, good and bad, suffer a major engagement decline barely 30 minutes in. That is where many gamers make the decision to invest the rest of their time, and if game completion is an objective, that is where developers need to invest in continuing to hook players along.
There’s plenty more to discuss (is a high early-game progress rate reflective of the ‘God of War’ school of design, which dictates building the first level last to make it the best? Is it reflective of early-game hand-holding?), but this article is long enough as is. I’ll leave you with a question: Why are most Hardest modes largely thoughtless, imbalanced damage increases and health decreases rather than intelligent remixes of enemies, engagements, and combat conditions?
Answer: Because no one ****ing plays on Hardest.
The opinions expressed here does not necessarily reflect the views of Game Revolution, but we believe it's worthy of being featured on our site. This article, posted originally on October 5, 2013 at 3:04pm PST, has been lightly edited for grammar and image inclusion. You can find more Vox Pop articles here. ~Ed. Nick Tan
|More On GameRevolution|