• 28Jun 09

  Do reviews really affect sales?

  There's one sentiment that I've seen quite a few times on the forums, and that's with respect to the effect of reviews on a game that was just released. When asked why they care about the fact that game X is getting poor reviews, they will often respond with something along the lines of, "It won't sell as well without good reviews, and that means we might not get sequels." At face value, this seems like an eminently reasonable suggestion - if people think that a game is bad, which they would discover through reviews, they would presumably be less likely to buy it. And yet, the persistent existence of shovelware - games that sell well despite being of low quality - seems to serve as an obvious counterexample to this thought.

  So, since I could not recall ever seeing any rigorous evaluation of the effect of reviews on sales in the current generation, I decided to take it upon myself to provide one for your benefit. Within this article, I plan to address the following questions:

  1. Do reviews have a significant effect on sales?;

  2. If so, how strong is that effect?;

  3. Which group of reviewers have the strongest effect on sales?; and

  4. Is that effect stronger or weaker on different consoles?

  I should issue a pre-emptive warning that this review will be making use of statistics, but I will try my best to explain in layman's terms everything that I'm saying, so I hope that you will keep reading even if you have no knowledge of statistics at all. I promise that you won't need any such knowledge to be able to understand my statements and conclusions. I have no idea how well this is going to be retrieved or if anyone is going to reply, but I figure I might as well give it a shot.

  If you're about to post "tl;dr" (which I can understand), you can click here to skip all of the data and jump straight to the conclusion. If, however, you would like to then dispute the conclusion, I would encourage you to read the article in full before doing so, in case any questions you might have are answered in the body of this article.

  So, to begin, I'd first like to explain my methodology, so you know what the data is that you'll be seeing shortly. In short, there are three sources for review scores that I retrieved: Metacritic scores, average scores from the top 5 review sites, and average GameSpot user scores. I defined "top 5 review sites" based on two criteria: the site's traffic ranking, as retrieved from Alexa; and whether or not the site was dedicated as a whole to video games. The latter criteria was imposed to filter out sites like Yahoo! Games, which is #2 in the world in traffic, but most of which traffic is likely not people curious about games. The top 5 were, in order, GameSpot, IGN, GameTrailers, GameSpy, and C heat Code Central. Critic review scores were retrieved from Metacritic; GameSpot user scores were, obviously, retrieved from GameSpot.

  In addition to review scores, I also retrieved total sales figures from VGChartz. While their weekly sales figures and sales figures for games just released are of questionable reliability, their historical and long-term sales figures are very much in accordance with the numbers released by the NPD Group, and as such, are quite reliable in comparison.

  For each of the top 50 best-selling games for the PS3, Xbox 360, and Wii, I retrieved both their total sales figures and the three review scores I mentioned above (Metacritic, the top 5 reviewers, and GameSpot user scores). I then compared the sales figures to the three review scores and asked the question of whether or not they are correlated - in other words, whether or not a higher review score is indeed correlated to higher sales.

  In order to make sure I was comparing apples to apples, there were two additional steps I applied to each game under consideration, as well. First, I weeded out all games that were bundled with peripherals, such as Rock Band or Wii Fit. Second, I estimated from the total sales figure how many copies had been or will be sold a year after release. Both of these steps ensured that the sales figures being compared were, in fact, legitimately comparable.

  Once I had comparable sales figures and the reviews for each game, I could begin testing for a correlation in each case, the results of which you can find below.

  Before I go on, however, there is one quantity that I should briefly explain that will come up, which is known as the R-squared value. When you have two sets of data that you're testing for a correlation, the R-squared value basically tells you how good the correlation is. A value of 0 means there's no correlation whatsoever; a value of 1 means that the correlation is perfect. A value of 0.25 means that about 25% of the relationship between the two quantities is explained by the setup. This isn't a precise definition, but it's good enough for the purposes of this space.

  Once you have the R-squared value, you can then test whether or not the correlation is significant. There is a certain value of R-squared (0.1303 in this case) above which you can basically say you have 99% confidence that the correlation cannot be explained just as a coincidence - and, thus, that there is a distinct relationship between the two quantities. Keep that figure of 0.1303 in mind as you see the results, as it's fairly important.

  So, without further ado, here are the results of this experiment:

  PS3

  Metacritic scores

  

  Average top 5 reviewer scores

  

  Average GameSpot user scores

  

  Remarks

  The most significant relationship here is the Metacritic score with an R-squared value of 0.2271, although just barely - the top 5 reviewer score has an R-squared value of 0.2232. These two are thus unlikely to have any significant difference between them, but both are well above the critical value of 0.1303, so we can conclude that there definitely is a very real correlation there. The GameSpot user score is less significant, with an R-squared value of only 0.1988, but it too is nonetheless above 0.1303. Thus, a higher score does indeed appear to be correlated in all three areas with higher sales.

  Xbox 360
  

  Metacritic scores

  

  Average top 5 reviewer scores

  

  Average GameSpot user scores

  

  Remarks

  Here, the most significant quantity is the top 5 reviewer score with an R-squared value of 0.1847, although, again, just barely - the Metacritic score has an R-squared value of 0.1763. Both are well above the critical value of 0.1303, so we can conclude that there definitely is a very real correlation there. The GameSpot user score, however, only has an R-squared value of 0.0824, which is not above 0.1303. Thus, a higher top 5 reviewer score and a higher Metacritic score appears to be correlated with higher sales, but a higher GameSpot user score is not correlated with higher sales.

  Wii

  Metacritic scores

  

  Average top 5 reviewer scores

  

  Average GameSpot user scores

  

  Remarks

  Here, the most significant relationship is the GameSpot user score, with an R-squared value of 0.1289. Lagging behind that are the Metacritic score and the top 5 reviewer score, which have R-squared values of 0.0997 and 0.0909, respectively. It is the case, however, that none of these are above the threshold of 0.1303, meaning that we cannot be 99% confident that that a correlation exists between a higher score and higher sales in any of these three areas.

  Conclusions

  The PS3 appears to be the console on which game sales are most connected to review score. On the Xbox 360, game sales are less tied to review score, but still definitely appear to be significantly correlated. On the Wii, however, there is a notable discrepancy with respect to the other two consoles - not only are no review scores significantly correlated with 99% confidence to game sales, but the most significant correlation exists between user scores and game sales, not between professional review scores and game sales.

  I believe that this underscores the fundamental difference between the Wii and both the PS3 and Xbox 360. From the data I have collected, it appears to be the case that the professional reviewers are most enjoying the best-selling games on the latter two consoles, whereas the general masses are most enjoying the best-selling games on the Wii. Thus, I believe that this gives at least a small amount of weight to Shigeru Miyamoto's statement in which he said that, when Matt Casamassina from IGN gave Wii Music a poor score, it was more the case that he simply did not want what Wii Music offered rather than that Wii Music was objectively bad.

  On the other hand, however, I do believe this also gives some weight as well to those who claim that many Wii owners are casual gamers who buy a game simply because the box looks fun, not because they know what they're buying - there are quite a few examples of games on the Wii that have sold well but have received negative reviews across the board from both professional reviewers and the general masses alike. An example of this would be Deca Sports, which has thus far sold 1.91 million copies to date, but which has a Metacritic score of 5.0/10, a top 5 reviewer average of 4.6/10, and a GameSpot user score of 5.1/10. This is a phenomenon found predominantly on the Wii - there were 4 games on the Wii that were in the top 50 yet had scores below 6.0/10 from all three sources, those being Game Party, Deca Sports, GT Pro Series, and Game Party 2. The only example of such a game on the other two consoles is Haze for the PS3.

  But, at the same time, perhaps the biggest conclusion that can be made is that on none of the three consoles did review scores actually account for that much of the sales. Professional review scores seemed to account for about 22% of the sales on the PS3, 18% of the sales on the Xbox 360, and 9% of the sales on the Wii. Certainly, one can say that it appears to be the case that reviews matter about twice as much on the PS3 and the Xbox 360 as they do on the Wii, but one could equally note that even on the PS3 it seems to be the case that almost 80% of the sales are essentially unrelated to the review scores that the games receive.

  So, at the end of the day, the story on all three consoles is about the same - people buy and enjoy the games that they want to play, no matter which console they own. And, after all, isn't that what it's all about?

  Other considerations

  As a final addendum, I should make a note of two things that I considered accounting for in the above experiment, but which I ended up not bothering. The first is with regards to the number of consoles that had been sold at the time a game was released. It occurred to me that it is surely more impressive to sell a million copies if there are ten million consoles in the install base as opposed to thirty million.

  However, there were two reasons why I did not account for this variable. The first is because any attempt to do so would be complete guesswork - I made a few attempts, but I had no idea whether or not the models represented reality at all. The second is because there is the second competing force, which is the fact that the number of competing games increases as well as time goes on. It is equally more impressive to sell a million copies if you have ten other games that people could alternatively buy instead of only three other games.

  Because it seemed to me that these two forces could conceivably cancel each other out, and because I had no idea how to properly account for them, I did not do so, and instead simply left the data as it was.

  Also, a legitimate question one might ask is with respect to how I could have estimated the number of sales a year following the release of a game. The reason is because the sales of a game over time tend to be exceptionally predictable in the long-term, and are logarithmic in nature (in layman's terms, that means that they rise very rapidly at first, but then quickly level off as time goes on). For this reason, I can just interpolate back to a year or extrapolate out to a year following the release of a game based on two sets of data: the release date of the game and the total number of sales at the present time. This quite accurately estimates the number of sales a year after a game's release.

  • Posted Jun 28, 2009 9:57 pm PT
  • Category: Editorial
  • 302 Comments
  • Tag
  • Print
  • Email
  • Facebook
  • Digg
• 8Jun 09

  A crash course on shovelware

  Shovelware.

  It is a word that sends shivers down the spines of hardcore gamers across the globe.

  Shovelware.

  

  OMG

  But, do we really have any idea what the word actually means when we use it? I submit that most have no idea at all. And thus, I would like to take this time in the hopes of educating people on the topic of what is - and, more importantly, what isn't - shovelware.

  So, to begin, what defines shovelware? Well, the dictionary isn't much help - it has a definition, but it's not the one we're looking for. Urban Dictionary's is a bit better, but is still sorely lacking:

  "Software that is hastily made, without proper testing, and 'shoveled' down consumers throats in order to make some quick cash."

  As always, however, one can rarely get the true sense of a word with all its subtleties from a single sentence. So, instead, let's look at an almost perfect case study: the game adaptation of WALL-E.

  

  Hey, baby. I'll lift your load anytime.

  The game was released concurrent to the hugely popular and financially successful Pixar movie on which it was based. Due to the large number of platforms on which it was released, it was developed in total by no less than four developers - none of whom are terribly well-known.

  The PC, Mac, and PS2 versions were developed by Asobo Studios, whose past games included Ratatouille, Garfield 2, and Sitting Ducks.

  Remember those games? I didn't think so.

  The Wii, PS3, and Xbox 360 versions were developed by Heavy Iron Studios, whose repertoire of video games is to date almost entirely movie-based spinoffs, their past games including Ratatouille, The Incredibles, and Up. The DS version was developed by Helixe, a developer who quite literally has produced nothing but movie tie-ins. The PSP version was developed by Savage Entertainment, a developer who, again, has a history of almost nothing but movie tie-ins.

  ...Yeah, when I say that they made this game for nearly every single platform that exists, I mean it.

  To date, the console versions of WALL-E have received a 6.0 on GameSpot - "fair" - and currently hold an average of 51 on Metacritic (50 for the Xbox 360 version). The handheld versions fared a little better, with their averages in the 70s. The game received praise from GameSpot for its well-constructed cutscenes, but was panned on account of its very simplistic gameplay, its worthless multiplayer, and the general ennui brought on by the experience of playing the game.

  The last paragraph of GameSpot's review in particular sums up the game very nicely:

  "The presentation is the only part of Wall-E that separates it from the glut of other platformers out there. The different gameplay mechanics offered by Eve and Wall-E provide some variety in this adventure, but the ideas are never fully fleshed out. This is still a fun game for anyone anxious to spend more time with the garbage-making robot, but there isn't enough depth or replay value to raise this above similar movie-licensed offerings."

  In short: the game isn't terrible, but it's boring; it's not falling apart at the seams, but it's not very satisfying and it doesn't feel fleshed out; it can be enjoyed by people who really want more of the movie, but it offers little other reason to play it above any other game.

  

  Oh. Joy.

  I mention all of the above because everything I have listed there is almost exactly what defines "shovelware". I called WALL-E an almost perfect case study in shovelware because it neatly encapsulates in one nice package almost entirely what is the essence of shovelware.

  To summarize, shovelware:

  - Is developed in a shorter time span than most games;

  - Is generally developed on every or nearly every gaming platform with fairly little serious consideration of the differences between them;

  - Is generally developed by a studio not known for their blockbuster hits;

  - Is generally critically panned, but not necessarily universally savaged;

  - Is shallow, stock, uncreative, risk-averse, familiar, uninnovative, and boilerplate to the extreme; and

  - Is developed in conjunction with or in the wake of a much bolder venture.

  One common definition that people give is to say that shovelware is designed solely to make money. And while this is true, it only tells part of the story, because, really, every game that exists is designed to make money. If studios did not intend to make money off of the games they developed, they would go out of business rather quickly. The differentiator between shovelware and non-shovelware is not so much the intention to make money, but the intention to make money not based on the merit of the game itself.

  To understand what I mean by this, consider the fact that the game adaptation of WALL-E has sold in total nearly four million copies across all its platforms. Now, ask yourself this: if this had been an identical game in terms of gameplay, but instead had been about Friedrich the Unknown Robot, would it have sold four million copies?

  

  ...Yeah. Exactly. The game was boring and stock, but it sold very well for two simple reasons: first, that it was released on every single current gaming platform there is, thereby maximizing its base of potential customers; and second, that the movie on which it's based is currently at 96% at Rotten Tomatoes and is sitting on top of a mound of over $500 million in revenue. In other words, it sold well because it was mass-produced across all platforms and then rode the wave of someone else's success. This is a critical component of shovelware: having no real redeeming qualities of its own, but nonetheless selling well due to being produced on tons of platforms and being tied onto the success of something else. It is, almost literally, shoveled onto its customers - the epitomic example of "quantity over quality".

  So, why am I saying all this? Because I have grown increasingly concerned at the sight of people declaring everything from Wii Music to, heck, even the Conduit as shovelware. Shovelware is not defined by poor sales. It is not defined by a low budget, or less-than-cutting-edge graphics, or a lack of effort on the part of the developers. And, similarly, shovelware is not - not - defined simply by the fact that one doesn't like it or isn't interested in it.

  Even the much maligned Carnival Games -

  

  - yes, even Carnival Games could have an argument pitched in its favor for why it isn't shovelware. For one, it was only made for the Wii and the DS, and for two, it didn't really have another runaway success that dragged it across the million-seller mark purely through its coattails. Many have attempted to argue that it attached itself to the success of the Wii itself, but if that were the case, then it would be difficult to explain why it now sits at over three million copies sold, while, say, Ninjabread Man (released at around the same time) is currently sitting at a paltry fifty thousand.

  The reason for its success, however, becomes more obvious when one peruses, say, the reviews buyers have left at Amazon. A sampling:

  "Carnival Games is the type of game that everyone should have for their Wii. It's great for solo play, or multi-player fun, and never becomes boring, with more than 25 games to choose from. It's extremely easy to 'pick up and play' for all ages, and it's good clean fun."

  "Talk about a surprise! This game offers exactly what you'd expect... mindless single player fun. Try it during a break from 'Metroid Prime Corruption.' With multiple players, though, this may be THE ultimate drunken Wii party game! .... It's a shame this has slipped under so many gaming site's rader and likely won't get the attention it deserves. This game fits the Wii perfectly... fun for everyone of all ages."

  "I started reading the game descriptions on this product months before it was released. I am a big fan of mini-games and this looked like it might be worthwhile. I suspected it might be a GOOD game, but I didn't expect it to be GREAT! Then I saw all the 5-star reviews and went ahead and bought a copy. Absolutely not disappointed! This is one of the best games for the Wii I've played to date!"

  These are all excerpts from the reviews at Amazon that are rated as "most helpful" - that is to say, the reviews that most closely approximated the experience that other buyers had upon buying the game. Compare these "most helpful" reviews to those for Ninjabread Man - which tear it to shreds - and the reason for the disparity in sales in evident. The thing to take away from this is simple: people like this game.

  I once heard a funny facetious definition of the term "puritan": "A person who is deathly, deathly afraid that someone, somewhere is happy." I think much the same sort of definition could be applied to a certain subset of gamers: "A person who is deathly, deathly afraid that someone, somewhere is enjoying a game that he or she isn't interested in."

  Really, the word "shovelware" does have its place - as I have shown above, it does certainly have a concrete meaning. But, much like the word "ironic", it has become increasingly overused to the extent that its very meaning has become threatened - which is precisely what spurred me to write this blog entry. Don't get me wrong - if you don't like or aren't interested in a game, that's perfectly fine. I am not by any means telling you that you should like the games I like or anything like that. Just, take a step back the next time you catch yourself about to label a game and ask yourself whether that label really applies, or whether you're just calling it that out of frustration. One is perfectly capable of not being interested in a game without feeling that there's something wrong with those who are interested in it. And dialog would be much more fruitful if everyone were mindful of that fact.

  Okay?

  Good deal.

  Now let's hug.

  

  • Posted Jun 8, 2009 11:28 pm PT
  • Category: Editorial
  • 100 Comments
  • Tag
  • Print
  • Email
  • Facebook
  • Digg
• 25May 09

  A tale of two religions

  It's been a while since my last blog, hasn't it? I suppose I've had a lack of inspiration for things to talk about. But that's changed, and for my first blog in a while, I'm going to be talking about something that I'm 100% sure everyone can get behind: religion.

  Ha ha ha!

  But anyways.

  I recently saw a thread in the off-topic forum that asked a question that I personally consider a very strange one, but which is nonetheless a pretty common one, and loosely paraphrased, it was the following: "Which has caused more harm throughout history: science or religion?" Predictably, there were a number of people who picked the obvious "enlightened" answer of "religion", some being more courteous about it, and a few going whole hog in declaring that the world would be much better if there were no religion.

  Those who know me fairly well will know that I'm pretty religious, so one can imagine how I was feeling while reading these messages. But despite my religiosity, I'm not here to preach to the choir and complain about such sentiments. Because, despite my religiosity, I understand them completely. For quite a while, I would have likely agreed with such sentiments, in fact. And having thought about it for a while, I believe that they and I are not so much in disagreement over the facts, but rather in disagreement over the idea of just what religion is.

  In short, I think that the ultimate problem in this situation is that there are two distinct and near-polar opposites that are both called "religion".

  On one hand, you have the more standard form that people are likely more acquainted with, which is the situation by which people attempt to learn facts about reality through their holy book. This is what I call the "being right" form of religion, the form by which people turn to their religion to learn facts about the world and to figure out the way the world works.

  But on the other hand, you have the form that people are sadly less acquainted with, which is the situation by which people attempt to learn instructions about the way in which they should behave in life. This is what I call the "doing right" form of religion, the form by which people turn to their religion to learn not the way the world is, but rather the way the world ought to be. This is the form of religion practiced by figures such as Desmond Tutu and Martin Luther King, Jr., people who found within their religion great inspiration and courage to do what they felt must be done in life.

  And, far from being two sides of the same coin, I feel that it may even be said that the extent to which one form is present is likely to be inversely proportional to the extent that the other is present. The reason for this is that, fundamentally, no matter how good someone's intentions are, one cannot hope to do what is right without first having correct information to act upon. And - and I am sure this will be met with controversy - those who practice the former version of religion tend to be, well, dead wrong in what they declare about the world. Between ideas such as that the Earth is the center of the universe and that disease is caused by demon possession, the track record of those attempting to glean factual information from their interpretation of their holy book is - and I mean no offense by this - terrible.

  From this last fact, many conclude that we ought to just throw out religion as a whole, because, so they say, you can't have one without the whole. But to this I say "not so fast". The idea that religion was never truly intended even from the start to give people factual information is nothing new. Even as far back as the fifth century, you have people like St. Augustine making remarks like this:

  "With the scriptures it is a matter of treating about the faith. For that reason, as I have noted repeatedly, if anyone, not understanding the mode of divine eloquence, should find something about these matters [about the physical universe] in our books, or hear of the same from those books, of such a kind that it seems to be at variance with the perceptions of his own rational faculties, let him believe that these other things are in no way necessary to the admonitions or accounts or predictions of the scriptures. In short, it must be said that our authors knew the truth about the nature of the skies, but it was not the intention of the Spirit of God, who spoke through them, to teach men anything that would not be of use to them for their salvation."

  To summarize what he's saying: the Bible is there to give us instructions; if we interpret it in a way such that it factually contradicts things we ourselves observe, then our interpretation of it is wrong, because that was never the point. This isn't some sort of modern-day cop-out, either; that quote dates to 408 AD.

  Unsurprisingly, Christians aren't the only ones who have said such things, either. Tenzin Gyatso, the current Dalai Lama, was quoted as follows:

  "If science proves some belief of Buddhism wrong, then Buddhism will have to change. In my view, science and Buddhism share a search for the truth and for understanding reality. By learning from science about aspects of reality where its understanding may be more advanced, I believe that Buddhism enriches its own worldview."

  So it seems pretty clear, therefore, that an inability to glean factual information ought not to deter someone from adhering to a religion, since the ultimate point is not to inform, but rather to instruct: to tell anyone who will listen how to best conduct themselves in life such that their life is fulfilling and a positive force in the world.

  It has also been claimed that this concern with doing right and with spreading love throughout the world is not religion, but rather spirituality. But pretty well all of us are familiar with what we know as the Golden Rule - that we ought to treat others as we want to be treated. It's an interesting fact to note, therefore, that this rule is present in the holy books of every single major religion in existence:

  Christianity

  "In everything, do to others what you would have them do to you, for this sums up the Law and the Prophets." (Matthew 7:12)

  Islam

  "None of you [truly] believes until he wishes for his brother what he wishes for himself." (13th of the 40 Hadiths of Al-Nawawi)

  Judaism

  "Do not seek revenge or bear a grudge against one of your people, but love your neighbor as yourself. I am the LORD." (Leviticus 19:18 )

  Hinduism

  "One should never do that to another which one regards as injurious to one's own self." (Anusasana Parva, Section CXIII, Verse 8 )

  Buddhism

  "He who, seeking his own happiness, does not punish or kill beings who also long for happiness, will find happiness after death." (Dhammapada 10)

  I could go on and list others, but you get the idea. It can perhaps be said that one can arrive at such conclusions without adhering to a specific religion - but, certainly, one can adhere to a specific religion and still come (and in fact be compelled to come) to such conclusions. The Golden Rule is perhaps the most culturally neutral moral statement in existence, and is understood to be sound advice by pretty well every single part of the world.

  Thus, I in fact both agree and disagree with those who assert that, for the world to progress, we must turn away from religion. I agree that the first kind of religion must fade away and that we must stop rejecting clearly observed reality in favor of one person's interpretation of a holy book. I do not believe, however, that that interpretation is the root problem. Rather, it is a symptom, with the ultimate cause being fear - fear of the unknown, fear of the world.

  Fear of the unknown is perhaps the strongest and most destructive of fears in existence - it is responsible for everything from racism to children's fear of the boogeyman - and I believe that it is this fear, and the inability to accept a lack of understanding and a lack of knowledge, that drives people to seek neat and tidy answers from preachers and religious leaders. To say that we must go beyond such things, therefore, is in essence to say that we must stop being afraid of this world in which we live, even if it can at times be scary, and even if our lack of understanding can at times be disconcerting.

  On the other hand, however, I could not disagree more strongly that the disappearance of the second kind of religion would be beneficial to the world. This form of religion, far from being borne from fear, is the ultimate acceptance that there are things greater than ourselves, that no human is an island, and that basing our own happiness on others' attainment of happiness is the only way that happiness can become a truly limitless resource. It is the antithesis of fear; it is an unbridled expression of love and a recognition that everything in the world only matters as far as it can increase the amount of happiness found therein. And, far from being the ball and chain attached to the foot of progress that is the other form of religion, this form of religion is the wings on the back of progress that turn otherwise useless knowledge and intelligence into the driving force on a journey to a better world - regardless of whether it comes from belief in God, belief in the beauty of nature, or simply belief in the human spirit.

  So, if you ever find yourself confronted with the question of whether or not religion is harmful, or whether or not we would be better off without religion, always remember to first ask the all-important question: which form of religion?

  • Posted May 25, 2009 2:20 am PT
  • Category: Religion
  • 36 Comments
  • Tag
  • Print
  • Email
  • Facebook
  • Digg

See Previous Blog Posts »