MEMBER BLOG

Did Game Revolution Kill The Secret World?
Posted on Saturday, July 14 2012 @ 02:13:36 Eastern

If you aren’t familiar with The Secret Worldcom scandal, the story unfolds as follows: Game Revolution reviewer Jonathan_Leack, in a manner that can only be described as “horribly underprepared for the consequences,” posts a review for MMO The Secret World, scoring the game as a 2.5 out of 5. The score gets entered into Metacritic as 50 out of 100 and is one of the first six reviews posted. The Secret World’s Metascore promptly sinks a whopping four points from a 74 to a 70.
 
On The Secret World’s forums, a thread is started to inquire about the low Metascore. Game Revolution’s uninflated review stands out as the worst score by ten points, prompting an inter-dimensional invasion of the review page. Secret Worlders create over 15 new Game Revolution accounts to attack the credibility of Jonathan, his prose, the review process, Diablo III—at one point cartoons are mentioned in a context so vile that my monitor catches fire.
 
For their part, the GR natives try to stymie the flow of insults by bathing everyone in acid. The situation is not improved:




Watching the thread erupt, I was struck by the absolute ire with which the TSW forum-goers attacked Jonathan’s review. Having already purchased the game themselves, why did they care? I’m not sure I’ve ever consulted the internet to determine my enjoyment of a game post-play, and I’ve certainly never requested that assessment from people who hate the things I love. But then, it wasn’t the review in its own right that first drew attention; it was the review’s effect on The Secret World’s Metascore… which makes even less sense, at least initially. If my personal experience with a game is a 90, and then the internet tells me it thinks the game is a 70, “Fuck, I’m wrong” is not the correct response.
 
But maybe TSW players care about the Metascore not for its influence on their perception of the game, but for its influence on everyone else’s perception. If potential inhabitants of Kingsmouth are opting instead for Azeroth ‘cause Metacritic, that’s less players in a community to which less future content will be pushed by a less well-funded studio. It’s layered selfishness, a black onion of the soul—but the rationale is sound. That is, assuming the Metascore determines a game’s financial success. Which it does not.
 
Right?
 
On March 15, 2011, THQ released Homefront, a military FPS designed and marketed to compete at the AAAA level with Battlefield and Call of Duty. That same day, the embargo placed on review copies of the game expired, flooding Metacritic with middling scores. Homefront’s Metascore plummeted from an 88 to a 72, and with it went THQ’s stock price, collapsing 21% from $5.94 to $4.69 over the course of 24 hours—a loss of over $85 million dollars in market value. The same game journalists who gave the 7 out of 10s that fueled Homefront’s decline criticized investors’ over-reliance on the Metascore, citing 200k preorders and 350k day one sales as reasons that the THQ would smash though its breakeven sales target of 2 million units and hit its 3.5 million unit goal. Homefront never came close, and the remaining THQ stockholders never saw their $85 million again.
 
Did investors know something game journalists didn’t? Could they have predicted the sales of Homefront months in advance using a single, aggregated number?
 
Almost unbelievably, the answer is yes.


Source: VGChartz and Metacritic

What we’re looking at is Xbox-only US sales data for 10 weeks of each major game’s launch from 2009-2011. Xbox and US data was chosen in order to remove the effects of install base and region.
 
There is a clear exponential relationship between Metascore and units sold, but you don’t have to take my word for it. We can use a statistical method called regression to create an equation that will predict, for every Metascore, what the expected sales volume will be. At a high level, a regression (specifically, a sum of squares regression) is just an equation created with the goal of minimizing the sum of the squared distance between each real data point and the data point that is predicted by said equation.
 
If you want to put your fist through my face for starting that last sentence with the word ‘just’, think of this instead: We create an equation that takes in a Metascore (x) and spits out a sales volume (y). For every Metascore, our equation will generate a single value for the expected sales of a game. We then look at the distance between that predicted number and all of the actual sales numbers for games with that Metascore. We square and sum all of these distances.  We then repeat the process for each Metascore until we have one giant sum of all the squared distances, or squared residuals, as statisticians and ******* bloggers like to say.

We then repeat THAT entire process for millions of equations, until we find the one equation that gives us the lowest sum of squared residuals (we actually just use math). This equation is the regression equation that best fits the data.



Above is our regression trendline. The formula in red is the equation that best fit the data: it is the equation of the red line to the left, which shows that as the Metascore increases, expected sales increase exponentially. The number in black is our R^2 (“R-Square”), one measure of fit for our regression. Read in English, this equation states: “43.56% of the variability in a game’s units sold can be explained by the variability in the game’s Metascore.” I will help level-set here by saying that for a one variable, non-time series sales data regression, 40% R^2 is pretty damn significant (go here for a near-perfect graphical explanation of R^2. The picture will also help you to visualize the regression process).
 
So, given that we’ve just shown a game’s Metascore is directly related to its sales volume, we can say with certainty that Game Revolution caused The Secret World to sell poorly by lowering its Metascore. Which would be fine, except for the fact that it’s CFW.
 
As in, Completely Fucking Wrong.
 
Correlation does not imply causation. And hell, even if it did, there are key sales variables that have less than one over infinity to do with the Metascore. For example: 



I said I selected the data to control for install base and region, but there are dozens of other factors that play a major role in determining a game’s sales—none of them related to Metacritic. Timing of the release relative to holidays, new franchise vs. established, genre, exclusivity, marketing spend, and publisher (which is more a measure of marketing dollar effectiveness than game quality) are all critical components of the game sales formula and are all completely beyond the purview of game reviewers.
 
In speaking of game reviewers, a Metascore is just that: An aggregation of individual reviews. Each reviewer comes with their own preferences for genre, mechanics, story, art—Metacritic cackles as it throws all of these viewpoints into a cauldron and boils them down into a type of Average Joe brew. This makes Metacritic a terrible tool for any one individual trying to figure out how they, with their own, unique set of preferences, will feel about a game, but a convenient approximation for an investor trying to determine how the ‘expected’ consumer will react to a game, regardless of whether the actual consumers get their own, personalized feedback from journalists, friends, demos, or forums.
 
Jonathan_Leack, one man with one set of preferences, posted an honest review about his experiences with The Secret World. He values smooth, skills-based combat, streamlined gameplay, and high production values. Unsurprisingly, Jonathan did not entirely enjoy the dodge-centric combat, abstract gameplay, and lore content over polish core of The Secret World. But because a Metascore is a reflection of its game and not the other way around, the only people his review will deter from playing The Secret World are the people who share his preferences, re: everyone who would not have enjoyed playing the game in the first place!
 
So rest easy, Secret Worlders—Game Revolution has taken nothing from you and asks only that you give nothing in return.
 
Kidding; we love you all. Stick around!