Editor's note: How much of an effect does Metacritic's aggregate scores have on games? I've always wondered about this, but I've never analyzed the data. Luckily, Rob has, and he's sharing his examination of Metacritic scores and sales for May 2009 with us. -Jason
The September 2009 issue of Game Informer published a feature discussing the merits of review aggregator Metacritic through the lens of Glen Schofield, former general manager for Visceral Games, which developed and released Dead Space late last year.
Central to Schofield’s complaint is a lone score of 65 that caused the Xbox 360 version of Dead Space’s aggregate Metacritic score to drop one point from 90 to 89. Schofield states that “the difference between an 89 and a 90 is a big-ass deal.”
Games journalist Mitch Krpata criticized the article, claiming that Game Informer was calling for reviewers to inflate scores in order to keep the Metacritic aggregate high. Whether or not Game Informer actually implied such is irrelevant to the observation that review score inflation would be at the detriment of the gaming public.
However, we all know that publishers use Metacritic scores to pressure developers by tying those scores to monetary compensation. Are publishers justified in using Metacritic in such a way? I decided to investigate the issue by tracking game sales and their accompanying Metacritic scores to find out the relationship between the two.
In order to have a manageable sample, I limited my study to games released during the month of May 2009. I did this because my source for game releases and sales data, VGChartz, only publishes the first 10 weeks of sales. May is the first month that all of the titles released had sales data for the full 10-week period.
I also feel that 10 weeks of sales data is enough to draw conclusions between the possible effects of Metacritic scores on game sales. The only systems that I included are the Nintendo DS, PlayStation 2, PlayStation 3, PlayStation Portable, Xbox 360, and Wii. Games needed to have sales data available in order to be included; in all, 31 games are in this study.
The above chart's representative of the study's full results. I’ve also highlighted several titles in order to give a sense of where certain games stand in the data.
We can see that game sales are only tentatively related to Metacritic scores. I say this because even if a game receives a high aggregate score, there's still the possibility of low sales numbers.
UFC 2009: Undisputed is the clear outlier here, with more than 1.6 million in sales and an aggregate score of 84. EA Sports Active and X-Men: Origins — Wolverine are the only other high-selling games — the former’s Metacritic score is 81, and the latter’s score is 63.
The lowest scoring game is Leisure Suit Larry: Box Office Bust, which got an 18. It's also one of the worst-selling games in the study, moving just over 30,000 units. The data also show that there are no poorly scoring games that also sold well (exceptionally or not).
From this chart, we can see that there's a clear upward slope of games with higher Metacritic scores being more likely to sell more games. But we can also see that many games with a high Metacritic score also sell poorly.
Many of the games studied sold fewer than 200,000 copies, so let’s zoom in on that data.
Boom Blox: Bash Party received an aggregate score of 86, yet only sold approximately 60,000 units. In contrast, Night at the Museum: Battle of the Smithsonian received a score of 52 but also doubled the sales of Boom Blox: Bash Party.
Again, we see several outliers, such as Sacred 2: Fallen Angel, which sold more than 190,000 units with a Metacritic score of 71. We also still see the same upward slope as the full data chart, just less pronounced.
The data also suggest that the majority of game releases sell less than 60,000 copies regardless of their Metacritic score, though a majority of the lowest-selling games had a Metacritic score above 60 (a total of 15 games).
Of the 10 games that sold over 100,000 copies, four of them were based on films (X-Men: Origins — Wolverine, Up, Night at the Museum, and Terminator: Salvation), two were sports related (UFC and EA Sports Active), and the rest were either remakes (Bionic Commando and Punch-Out!!), sequels (Sacred 2), or exclusives (Infamous). This suggests, at least to me, that there are more powerful factors that influence game sales than Metacritic scores.
Are publishers correct to use Metacritic scores as a primary factor in understanding how well a game sells? The data does suggest that no poorly scored games are going to also sell well; however, the data also suggest that a high Metacritic score isn’t a guarantee of high sales, either.
Additionally, most games appear to fall into the top left corner of moderate-to-high Metacritic scores and relatively low (less than 60,000) unit sales. While the profits from such sales are nothing to sneeze at, it’s generally not enough for games with budgets in the tens of millions.
My opinion from the data is that Metacritic scores are not as important as publishers would like to believe them to be. Other factors have a bigger influence on game sales. The data from this study's limited in providing answers to what those other factors may be and how much influence they may or may not have on game sales.
Methodology: Game release dates and sales data were collected from VGChartz. Only games that had sales data for the 10 weeks were included, with the exception of three titles (Raiden Fighters Aces, Magicians Quest: Mysterious Times, and Challenge Me: Brain Puzzles. For the purposes of this study, these games were assumed to have had zero sales for weeks nine and 10. All three games sold less than 25,000 copies combined.) All sales data used is based only on U.S. sales.
I combined cross-platform game sales and averaged Metacritic scores between platforms to arrive at one average Metacritic score for all versions of a game. Nintendo DS, PlayStation 2, PlayStation 3, PlayStation Portable, Xbox 360, and Wii were the only platforms represented because sales data for other platforms was unavailable.
I used May 2009 as a sample in order to keep the data collection manageable. May was also the first month that all games released had 10 weeks of sales data available. A more thorough investigation could be completed by compiling all data from the first six months of 2009. Such an undertaking may be the subject of a follow-up study.