GamesBeat Metacritic Scores vs. Sales: 2009 Holiday Season Analysis April 28, 2010 4:56 PM Rob Savillo 0 This post has not been edited by the GamesBeat staff. Opinions by GamesBeat community writers do not necessarily reflect those of the staff. Recently in GamePro, Julian Murdoch examined the reviews aggregator site Metacritic. His story opens with a rather startling observation from Activision Vice President of Marketing Robin Kaminsky at the 2008 Design, Innovate, Communicate, and Entertain (DICE) Summit. The opening slide to her presentation read: "For every additional five points over an 80 percent average review score, sales may as much double." Incredible! If true, game sales must increase exponentially as they score above an aggregate of 80 on the website. Surely, an employee of Activision — let alone a VP of marketing — must have solid data to back up such a bold declaration. But I'm the skeptical type, so I did a little of my own research. I’m here to tell you that Kaminsky's statement is a load of horse shit, and that publishers' insistence on a correlation between scores and sales is unfounded. Murdoch’s article, which is well worth reading, focuses not on Metacritic’s influence (or lack thereof) on sales, but on the connection that publishers make between scores and sales. In other words, he investigates the shady practice of tying developer compensation to review scores. My goal, however, is to see if any meaningful correlation between Metacritic scores and unit sales exists at all. When I previously studied this issue, I concluded that the former have virtually no influence on the latter. Critically acclaimed games, like the Wii’s Boom Blox: Bash Party (which received a Metacritic score of 86), sold poorly, while much lesser-scoring games, such as Night at the Museum or Terminator: Salvation, sold significantly more. My sample size for that examination was small — I used May 2009 releases and 10 weeks of their sales data. To investigate this matter further, I decided to collect a much larger, more representative sample. My analysis for this article includes everything that came out between October 1 and December 31, 2009. For each title, I collected 10 weeks of sales data as it was posted on VGChartz. Any releases that did not have the full 10 weeks and a Metacritic score were not included. Out of 364 titles (including all system ports between the Nintendo DS, Nintendo Wii, PlayStation 2, PlayStation 3, PlayStation Portable, Xbox 360), 218 made the cut. PC, iPhone, and digital releases were not included because of the lack of sales data. Read on to page two for the breakdown. First, I’ll analyze the data in the exact same fashion I did back in September. I combined cross-platform game sales and averaged Metacritic scores between platforms to arrive at one score for all versions of a title. Click for a larger view We have a few clear outliers — Modern Warfare 2, New Super Mario Bros. Wii, and Assassin’s Creed 2. These three were the only titles to sell more than three million copies (approximately 9.5 million, 5 million, and 3.3 million, respectively). They were also some of the highest scoring games (94, 87, 84, respectively). We can’t really tell much from the data by looking at this chart, so let’s remove the outliers from the analysis. Click for a larger view Here we get a slightly clearer picture, but we still have this giant mass of games in the top-left corner of the chart. Like my previous study, this data gives a similar impression — a few games sell well and generally score above 60, while most others sell poorly regardless of their score. Again, no games score extremely low and sell very high. A few interesting examples, though, are in this chart. Tony Hawk: Ride (PS3, Xbox 360, and Wii) sold about 943,000 copies with an averaged review score of 46. On the other hand, the Xbox 360 exclusive Forza Motorsport 3 sold similarly (about 960,000 units) but scored twice as high — a 92. What’s going on here? Another comparison that gives me pause — the PSP version of Grand Theft Auto: Chinatown Wars sold more than 120,000 copies with Metacritic score of 90. But Rogue Warrior, which appeared on the PS3 and Xbox 360, sold nearly the same with an aggregate score of 28. Yes, you read that right. The cluster of games in the top-left corner deserves closer examination. It looks like most of these titles sell less than approximately 500,000 copies, so let’s zoom the chart in there. Click for a larger view The data shows no clear trendline. We have a large scatter of points between scores of 40 and 80, with few outliers. Again, I’ll make another comparison to GTA: Chinatown Wars — look at Star Wars: The Clone Wars: Republic Heroes, which sold almost four times as many copies with an averaged score of 45 — fully half that of Chinatown Wars. But it may not be fair to combine cross-platform data like that and make valid comparisons — handheld and Wii versions can vary greatly from their PS3 and Xbox 360 counterparts. Certainly, that data is interesting, but let's review scores and sales for each system. Click for a larger view Again, the full chart doesn’t tell us much, other than that most games are going to sell less than a million copies, regardless of their Metacritic score. We see the same outliers, too. Let’s zoom right in to those that sold 500,000 copies or less. Click for a larger view Once again, no clear trends come out of the data. Each system’s data points scatter widely, with no clear correlation apparent between sales and scores. Interesting to note is that PSP games sell especially poorly (LittleBigPlanet is the handheld’s highest selling game with approximately 260,000 copies). Also interesting is the sharp drop in quality for Wii games as they sell more copies. As we go beyond 300,000 units sold, Wii games drop from a Metacritic score of 72 to a score of 48 (LEGO Indiana Jones 2: The Adventure Continues, NERF: N-Strike Elite, Hasbro Family Game Night 2, Tony Hawk: Ride, Just Dance, respectively). I don’t think this is a trend of any sort, but it’s the only system to have such poorly scoring games also sell well. Looking at the top five games in both sales and scores, we can see problematic examples. High scoring games such as Half-Minute Hero (PSP), Pro Evolution Soccer 2010 (Wii), and Might and Magic: Clash of Heroes (DS) all sold terribly low — 34,000, 57,000, and 40,000 copies, respectively. And a few high sellers that far outpaced the aforementioned titles, Assassin's Creed: Bloodlines (PSP), Just Dance (Wii), and Pokémon Mystery Dungeon: Explorers of Sky (DS), scored poorly — 63, 48, and 53, respectively. Below are tables with each system's top five games ranked by Metacritic scores (left) and unit sales (right). PS3 Rank Total Sales Score PS3 Rank Total Sales Score Uncharted 2: Among Thieves 1 1,118,574 96 Modern Warfare 2 1 3,310,903 94 Modern Warfare 2 2 3,310,903 94 Assassin's Creed II 2 1,285,815 91 Assassin's Creed II 3 1,285,815 91 Uncharted 2: Among Thieves 3 1,118,574 96 God of War Collection 4 554,536 91 God of War Collection 4 554,536 91 FIFA Soccer 10 5 280,984 91 Dragon Age: Origins 5 455,755 87 PSP Rank Total Sales Score PSP Rank Total Sales Score GTA: Chinatown Wars 1 123,695 90 LittleBigPlanet 1 260,086 87 LittleBigPlanet 2 260,086 87 Assassin's Creed: Bloodlines 2 219,983 63 Half-Minute Hero 3 34,287 84 Star Wars Battlefront: Elite Squadron 3 149,392 63 FIFA Soccer 10 4 97,475 83 GTA: Chinatown Wars 4 123,695 90 Tekken 6 5 70,748 82 Yu-Gi-Oh! 5D's Tag Force 4 5 121,150 70 Wii Rank Total Sales Score Wii Rank Total Sales Score New Super Mario Bros. Wii 1 5,042,122 87 New Super Mario Bros. Wii 1 5,042,122 87 DJ Hero 2 240,915 87 Wii Fit Plus 2 1,967,654 80 Pro Evolution Soccer 2010 3 57,190 82 Mario & Sonic at the Olympic Winter Games 3 893,648 72 Wii Fit Plus 4 1,967,654 80 Modern Warfare: Reflex Edition 4 549,603 76 EA Sports Active: More Workouts 5 194,325 80 Just Dance 5 470,787 48 DS Rank Total Sales Score DS Rank Total Sales Score The Legend of Zelda: Spirit Tracks 1 788,519 87 The Legend of Zelda: Spirit Tracks 1 788,519 87 Might & Magic: Clash of Heroes 2 40,516 86 Mario & Sonic at the Olympic Winter Games 2 520,568 76 Space Invaders Extreme 2 3 43,989 82 Bakugan: Battle Brawlers 3 501,933 60 LEGO Rock Band 4 107,884 80 Pokémon Mystery Dungeon: Explorers of Sky 4 348,233 53 Mario & Sonic at the Olympic Winter Games 5 520,568 76 Drawn to Life: The Next Chapter 5 309,289 72 X360 Rank Total Sales Score X360 Rank Total Sales Score Modern Warfare 2 1 6,232,345 94 Modern Warfare 2 1 6,232,345 94 Forza Motorsport 3 2 963,155 92 Left 4 Dead 2 2 1,938,107 89 Assassin's Creed II 3 1,936,410 91 Assassin's Creed II 3 1,936,410 91 FIFA Soccer 10 4 384,540 90 Forza Motorsport 3 4 963,155 92 Left 4 Dead 2 5 1,938,107 89 Dragon Age: Origins 5 938,898 86 From this data, what can developers and publishers expect after a game's release? Based on Metacritic’s own scoring system, I split the data into four categories: universal acclaim (a score of 90 or more), generally favorable reviews (between 89 and 75), mixed or average reviews (between 74 and 50), and generally unfavorable reviews (49 or less). I decided not to use "overwhelming dislike" because no games scored that low in my sample. Overall, most titles — 58 percent — received mixed reviews. Additionally, most sold less than 100,000 units (62 percent). Even more depressing, a plurality of games — 36 percent — sold less than 100,000 copies and received mixed reviews. Twenty percent of all games studied sold between 100,000 and 500,000 fell into the same scoring category. Only three percent of all games analyzed sold more than 1 million copies and received Metacritic’s most coveted rating — universal acclaim. Click for a larger view Includes sales data for games studied without a Metacritic score. Click for a larger view. I suppose that Kaminsky gave herself an out by qualifying her statement with “may.” If what she proposes were actually true, we’d expect to see the above scatter charts’ sales data curve upwards exponentially above a Metacritic score of 80. In fact, what we actually see is most games lingering in the top-left corner of the chart (i.e., low sales, moderate scores) with a few runaway titles garnering millions in sales. My analysis demonstrates that no link between Metacritic scores and sales exists. So why do publishers insist on such a connection? I can’t really say. That reviews scores would influence sales does feel intuitive. Perhaps publishers also feel reassured that they've based their decisions on a quantifiable number. But I won't pretend that I have all the answers. I don't really know why or why not a particular game sells well or poorly. My data isn't equipped to answer that question, either. What I do know is that no correlation between Metacritic scores and unit sales exists, and I think it's foolish for publishers to tie monetary compensation to such scores.