Reality Check: Reviewing Games Right Is Hard

In stores next week. In our hands for review soon, hopefully.

For starters, contrary to what you might think, we don't always get copies of games far enough in advance of their arrival on shelves to facilitate release-date reviews. At the time of this writing, we've just received PlayStation 3 and PC copies of Brink (in stores today), we've yet to receive MX vs. ATV Alive (also in stores today), and it's looking like we might not get our hands on The Witcher 2: Assassins of Kings (in stores next week) this side of the weekend. Historically, receiving a game too late to turn around a review ahead of its release date was often indicative of a publisher's lack of confidence in its product. That's still the case occasionally, but increasingly even great games aren't being sent out for review in a timely fashion. In fairness, the fact that we insist on reviewing finished games on retail hardware probably doesn't help matters, but that's the only way we can be sure that we're reviewing exactly the same game that's going to be sold in stores. The unacceptable alternative is to base our reviews on code that isn't 100 percent final and that, more often than not, arrives in the mail with a list of caveats detailing problems that will really, totally, honestly be fixed before the game ships. Sadly, even our policy of reviewing retail copies of games doesn't always ensure that we have the same experience you will.

In September 2009, we posted our reviews of the Xbox 360 and PlayStation 3 versions of NHL 10 and noted that the latter's online play suffered from noticeable lag. That review was accurate when we posted it the evening before the game's release, but mere hours later, the PS3 servers had been optimized, and the multiplayer was silky smooth. We opted to change our original review on that occasion for reasons that are detailed in this blog post. That situation was a first for us, but the current prevalence of day-one patches (sorry, updates) means that it's not uncommon for games to change significantly after they're pressed and packaged for retail but before they go on sale. In October 2008 we added an update to our review of Fable II after testing the cooperative play that was added via an update on day one, and just this week we decided to delay our Brink review so that we could determine for ourselves the significance of a promised day-one patch.

Despite the fact that Brink's release date was brought forward by a week not so long ago, apparently because production on the game had finished earlier than planned, the arrival of our boxed retail copies was preceded by an e-mail detailing a number of known problems that would be addressed by an "automatically-applied day one update on release." These included "sporadic visual glitches at a distance," "texture pop-up," and "minor networking issues." Sure enough, all of these issues and more were evident when we started playing. Unfortunately for Bethesda and Splash Damage, publisher and developer respectively, the patch didn't hit until after the review embargo was up, so authors of the first reviews to show up online weren't able to review the "finished" version of the game. Indeed, according to Paul Wedgwood, Splash Damage CEO and game director, talking on the GameSpot UK Podcast earlier this week, "Some early US journalists were playing on a really broken, horrible build." He went on to say, "I think it had broken textures, broken network code, and broken AI and stuff," hopefully unaware that reviewers in the US were sent the exact same discs currently being sold in stores more than a week ago.

Significant (and purportedly significant) day-one patches are always a headache, because by choosing to wait for them we end up looking like we're "late" or "slow" (as opposed to "thorough" or "diligent") with our reviews. We face a similar problem anytime we review a game with a significant online component. Most publishers are good about sending multiple copies of games and organizing online sessions for us to participate in with other reviewers, but these sessions aren't always played on retail servers (we had to manually configure the DNS settings on our retail PS3 in order to test SOCOM 4: U.S. Navy SEALs, for example), and even when they are, it's impossible for us to know how well the game's servers will perform when thousands of players are online simultaneously, based on time spent with fewer than a hundred. Furthermore, even waiting until day one to test the game online (which we've done on occasion) is an imperfect solution, because there are generally a lot more players hitting the servers a lot harder during the first couple of days than there are a week later.

These problems are symptomatic of a much larger issue, which is that games are no longer static pieces of entertainment; they're not finished and then forgotten. Rather, many of them evolve over time, and in the process, they sometimes render reviews posted at the time of their release inaccurate. The nature of games has significantly changed, but the way that we review them, for the most part, has not. The After the Fact updates that we introduced on GameSpot in 2008 afford us a way to add updates to reviews that are in need of a refresh, but perhaps what's really needed is an entirely different approach.

Reality Check is GameSpot's recurring editorial column. Each week, members of the GameSpot editorial team sound off on current gaming events as well as various topics that surround the gaming industry.